Joomla 1.6 caching explained

Joomla 1.6 caching for developers

Table of contents

1. Choosing a host for your Joomla site
2. Installation
3. Setting up and configuring the website
4. Running the website
5. Developer’s checklist
6. What to do if your site gets hacked
7. Further reading, other resources
8. Acknowledgements

Choosing a host for your Joomla site

First of all, you should find a proper hosting provider. You could ask others in the Joomla community for their recommendations, and you could check out the shortlist of hosting providers on the Joomla Resource Directory. These have at least gone through a manual screening process where their server setup has been checked for the most common pitfalls.

If you’re considering a host, here’s a brief list of requirements that you can run by your hosting provider to see if they fit the bill:

1. Secure communication

   Make sure your host supports secure connections. It will cost you around $30/year to get a 256-bit SSL certificate to secure your backend. If you need to transfer files, make sure that SFTP or SCP is an available option to you, instead of having to use plain FTP.

2. PHP configuration

• As with any software: Don’t use outdated versions, like PHP4
• Disable potentially harmful PHP functions
• Use open_basedir (with care, see comments after this article by Nicholas)
• Turn magic quotes off
• Don’t register globals
• Allow users to set local php.ini files

3. Apache configuration

• Enable .htaccess
• Enable mod_rewrite
• Enable mod_security
• Enable suPHP

Installation

4. Connect securely

   Use SFTP or SSH when you connect to your server, and do not store your password in for instance your FTP client. This of course also goes for any subsequent connections to the server after you have finished your install.

5. Follow your own best practices

   If you have already set up one site where you follow all the basic security measures you want to follow, and you have already implemented the security extensions of your choice, there is no need to reinvent the wheel: Use a backup tool like Akeeba Backup or something similar, take a backup of the site and use that backup as a starting point for your new site.

   Kudos to @hermanpeeren for the tip!

6. Do not use the standard DB prefix jos_.

   Many SQL injection attacks are based on the assumption that your database tables are named “jos_”. In case you did use jos_, you can still change it with a simple PHP script.

7. One database user per installation

   Do not give the same database user access to multiple installations, and do not use the server’s root user for your database connection.

8. Consider db user password quality

   Use a long and complicated password for your database, and when installation is done, for your login. It’s too easy to decrypt an MD5 or SHA1 hash these day. There are even webapps that help you do it. If someone gets hold of your database of hashed passwords, it’s only a matter of time before they are broken. There are lots of good tips for password quality in this previous article I wrote, “How to create—and remember—a different password for every single login“, and there are also a lot of good tips in the article’s comments.

   In general, when it comes to passwords: Don’t write them down anywhere, and if you have to, at least store them securely.

9. Don’t install sample content

   Sample content identifies you and your installation, and makes you easy to find and target. Don’t install sample content on what is to be a live website, because forgetting to remove absolutely all sample content is just too easy.

Setting up and configuring the website

10. Use correct file- and directory permissions

    Avoid 777 of course. It seems this can’t be repeated enough times though. Extensions like suPHP will take care of setting proper permissions, so talk to your host and make sure they support it.

    This will also make the need for the Joomla FTP-layer redundant, so you won’t neet to use your FTP username and password anywhere.

11. Uninstall stuff you don’t need

    Brian Teeman has a saying: “Nothing appears on my website that I didn’t put there“. Uninstall core components, modules, plugins, templates and other extensions that you’re not using, including templates and even images. Extensions on your website are all potential security liabilities, either because they add security holes or because they help attackers identify potential security holes.

    Observe for instance that installing the JCE WYSIWYG editor will also install TinyMCE, as it is bundled with JCE. Joomla comes with TinyMCE installed as it’s default editor, so then you’ll have two TinyMCE installations.

    Some core stuff can’t be uninstalled through the normal uninstaller. This is because they are marked as “core” elements in the database. In the components, modules and plugins database tables, there are fields called “core” which is set to 1 for these extensions. Change this value to 0 to make them possible to uninstall.

12. Global configuration setup

    1. Disable registration/login

       If you don’t want people to register and/or login on the frontend, make sure you disable the functionality.

    2. Disable XML-RPC

       Disable the XML-RPC-server if you don’t need it.

    3. Replace default Joomla meta information with your own data

       Make sure you replace, or at least replace, the default metadata. These will show up in your sourcecode and in Google.

    4. Use SEF urls

       Regular URLs gives away the fact that Joomla is running your system. But then again, so can a lot of other things. Like for instance a template graphic, a static file, a code structure. Changing from regular to SEF URLs won’t render the regular URLs useless either, so this is not a very strong security measure. Make sure you also enable the .htaccess-file, see below.

    5. Move logs and cache outside the site root

       Change the log and cache paths in the administrator to a folder outside the site root so that they can’t be accessed through a web browser, or protect the folders in some other way.

13. Remove generator information

    Remove the generator tag from the head-section of your website.

14. Delete the default user account

    The userid 62 and the username “admin” represents a weakness, but there’s no easy way to replace these during installation. In the administration, the easiest thing you can do is to add a new super administrator, log in as the new super administrator, then demote the admin user to a lower usergroup and then delete the user all together. As our greek friend @nikosdion demonstrates, you can also fix this and add users below id 62 with some easy SQL.

    Here’s some quick SQL to help you change the user id and username of the default account right after installation:

    -- Change 3 in both queries with the userid you want
    -- Change jos_ with your own db prefix
    -- Change yourusername with your new username
    UPDATE jos_users SET id=3, username='yourusername' WHERE id=62;
    UPDATE jos_core_acl_aro SET VALUE=3 WHERE VALUE=62;

15. Enable the .htaccess-file

    The default .htaccess-file has some basic security built in. In Joomla versions earlier than 1.5.15, you should also make sure it restricts access to critical files. In versions newer than 1.5.15 (1.5.20 is the latest version at the time this writing), you still have to uncomment parts of the .htaccess-file to activate this security measure. 

16. Enhance the .htaccess-file

    Nicholas K. Dionysopoulos (that’s @nikosdion in twitterish) also has a nice Master .htaccess-file with lots of other goodies included, that’s really worth a peek. For instance: Restrict directory listings, force https on certain pages, blocking of some common exploits, file injection protection, fingerprint attack blocking, and more.

17. Encrypt communication, or at least the login

    A nice article by Herman Peeren highlights the dangers of not logging in to your site with SSL, and suggests that if you can’t set up SSL you at least look into an extension called Encrypt configuration.

18. Make fingerprinting your site harder

    One of the easiest ways for an attacker to decide if your site is a potential Joomla! target is to perform a rudimentary visual fingerprinting. Both Joomla! and PHP comes with many static resources, special features and even easter eggs that can make it easy to identify what software you’re using, and what versions they are.

    There is a great post in the October 2010 Joomla Community Magazine about this: “Only a ninja can kill another ninja“, by @nikosdion.

19. Choose your extensions wisely

    1. Avoid extensions with encrypted sourcecode

       You can’t fix bugs in encrypted sourcecode yourself, and you’ll be left at the mercy of the extension producer should something bad surface. This is mostly a personal preference of mine, as I like probing around in people’s code both to learn and to get a general feeling for the actual quality of the software I’m running.

    2. Avoid dead extensions

       If a extension’s homepage is returning a 404 error, or hasn’t been updated in ages, it’s a good indicator that you should check out if this extension is still being maintained or not, before you decide to install it.

    3. Avoid legacy extensions

       Stick to extensions with fresh 1.5 compatible code. Joomla 1.0′s EOL has passed now, over a year ago, time to move on.

20. Add extra layers of security

    1. Protect /administrator with extra measures

       You could require an extra password to access /administrator, or even restrict access to certain IP addresses. Some security plugins, like jSecure can help you accomplish stuff like this too. (thanks to @nikosdion again)

    2. Consider installing security extensions

       Lots of opportunities on the JED in the categories site protection, site monitoring and login protection. Many of these will help you with the items in this list.

    3. Monitor your site for unwanted changes

       Lots of tools can be used here, for example there is a Windows tool out there called Akeeba SiteDiff, which you can use with Akeeba Backup, and which will compare the state of your site with the last known good state. Read more about Akeeba SiteDiff.

    4. Add a custom ACL solution

       Joomla 1.5′s default ACL is not very restrictive, and you may be giving out more access privileges than you’d like. Brian Teeman has a nice table showing who gets to do what exactly. Consider adding a custom ACL solution for better access control.

21. Avoid exposing information

    Disable the display of errors on live sites, so that system paths and server information is not shown. Set display_errors to 0 in a local php.ini for instance, or use php_value in a .htaccess-file.

22. Don’t send out username/password in plain text e-mails

    Sending a plain e-mail is like sending a postcard; it’s not wise to include anything sensitive in it. Joomla does this by default (!) each time you add a new user to your users list, or when a user signs up.

    One way to avoid this is to override the language files; the text that goes out in the account creating confirmation e-mails is located in the language file, so this can be tweaked to not include the password.

    In Joomla 1.6 you can also override a single language string without having to create a whole language pack, which makes this easier. There is also a plugin for 1.5 called Translation Override, which may prove to be useful for this purpose. (thanks to @jlleblanc, @nikosdion and @ot2sen for the tips)

Running the website

23. Stay updated

    There’s one newsfeed for Joomla Security News and one feed for news about vulnerable extensions.

    You should use the information to avoid insecure extension, because you can bet that those looking to hack your website are also keeping up to date with news on new vulnerabilities. – Read the result of Brian Teeman’s “Investigation of Exploits” from August 2010. – Read the vulnerable extensions list in the docs wiki, and subscribe to it’s feed.

    Note that you can also subscribe to changes on a wiki page like the one above, to be notified whenever it changes. If you go to the vulnerable extensions list and register an account and log in, you will get a button in the top menu that allows you to subscribe to changes on that page.

24. Install security patches immediately

    Pay attention to mailing lists announcing insecure extensions and updates to extensions you’re using, including Joomla itself of course.

25. Backup your site regularly

    If your server host does not do backups for you, you could backup your site to the cloud, or make good use of a backup extension.

    Make sure to test that the backup works and always store the backup off-site.

Developer’s checklist

26. Use an IDE

    IDEs can help considerably with code quality, development and testing.

27. Use a versioning system

    Stuff like SVN and Git will be a lifesaver the day something is hacked or modified without your knowledge, or goes to hell in another way. It also makes it easier to maintain and roll out different versions of your software. Use versioning even if you’re the only one doing development, it’s well worth it.

28. Test locally, deploy globally

    Run different configurations for your test sites and live sites: Stuff that gives away information should never be done where it’s visible to the public, in particular debugging. Errors should not be shown on a live site, especially not those who disclose paths or system information. As an extensions developer though, all you can do here is probably just advise your customers. This type of setting has to be done on the server.

29. Use CSRF-protection

    http://docs.joomla.org/How_to_add_CSRF_anti-spoofing_to_forms

30. Don’t chmod files or folders to 777!

    Not only is this a bad idea security wise, it also tells people reading your code that you’re lazy at best, and furthermore that you don’t know what you’re doing.

31. Filter data

    Filter data the user inputs and escape outputting of user-provided content, which is especially useful for preventing SQL injection and XSS attacks. Read more about this and see examples.

32. Keep your codebase small

    Reuse previously road-tested code where you can, and stay DRY (Don’t Repeat Yourself). Less code equals less potential security risks and less code maintenance costs for you.

33. Use Nooku Framework

    If you’re creating a Joomla extension, you should consider using Nooku Framework, which reduces your codebase with around 80% and does a lot of things automatically, where you in Joomla would have to write code manually. Directly related to security, Nooku gives you automatic CSRF protection (automatic form tokens and token control), automatic data filtering and soon also automatic output escaping by default.

34. No executable code in class files

    In your class files, stick to one class pr file and leave no executable code.

35. Use _JEXEC

    use a _JEXEC test in files, especially templates or files with executable code, to make sure the file is loaded from within the application. Files with just a PHP class in them do not need this.

36. Keep names and versions hidden

    Don’t display extension names and versions in the code or on your site. “Powered by Superapp version 1.5″ on page footers that use your application, combined with a listing of Superapp version 1.5 being vulnerable on the list of vulnerable extensions, makes your clients easy targets. At least don’t display this info until Joomla gets a better upgrading system so that people can update outdated extensions in a painless manner.

    Worth reading in this context: New Software Notifications in Google Webmaster Tools.

What to do if your site gets hacked

37. Take your site offline

    If malicious code has been injected into your site, you’d not want others to get their computers infected or compromised. For tougher cases: Add an IP filter so that only your IP-address can access your site while your fixing it.

    Remember: Offline does not mean unavailable

    Setting a website in maintenance mode is merely a visual thing that indeed will execute all plugins and component requests, but merely render the offline template. So if someone has found a loophole to get into your site, and you want to shut them out while you fix the problem, setting the site in maintenance mode will not help you.

    Nicholas Dionysopoulos published a blog post with information on this subject which is useful. He outlines one possible way around this issue, where he uses .htaccess to redirect all request to offline.html directly, bypassing Joomla completely. This means you have to have an offline-page of pure HTML to serve too of course. He also adds an exception so that you yourself can access the Joomla page from a specific IP-address.

    Read the original blog post for a more in-depth explenation and code sample.

38. Find other people with similar errors

    First thing to do would be to search the Joomla forums or search the web to try to find scenarios similar or identical to yours. This can save you a lot of time in the recovery phase, since you may benefit from other people’s findings.

39. Make sure your computers haven’t been compromised

    Malicious software on any computer with is used for accessing the site or it’s FTP accounts for instance, can be used by an intruder to gain access to your online user accounts. Use a scanner software to scan for malware, trojans, viruses, spyware, etc. A list of available software packages can be found in the Joomla documentation under “Local security”.

40. Verify that you have the latest versions

    Check that you have the latest Joomla version and the latest versions of your extensions, and that you’re not using any extension versions with known vulnerabilities. See lists of vulnerable extensions on the Joomla documentation wiki and in the Joomla security newsfeed.

41. Locate modified files

    Log in to your server and use the find command to find out which files have been changed recently.

    This command will list all files in the current folder where the file modification time (mtime) is less than three days old:

    find . -mtime -3

    This command will list all files in the current folder where the last change of file status (ctime) happened less than three days ago:

    find . -ctime -3

    This command will list all files changed at least one day ago but less than three days ago:

    find . -ctime +1 -a -ctime -3

42. Find out what’s changed on your server

    If files have been modified on your server, or files have been uploaded for instance, you can check the timestamps on those files to find out when the attacker was on your site. This is typical in the case of sites being defaced or malicious code being injected somewhere. Most of the time, the attacker will have gained access to your site shortly before modifying or uploading files to it. By checking your access logs for the period around that time, you may find some clues as to how the attacker gained access and which IP-address the attack came from, thus enabling you to track down things more easily in the logs.

    You can also try searching for suspicious POST requests made to for instance non-form addresses on your site.

    Use any information gained from this to block out further attacks and patch weak spots in your system.

43. Check your crontab/task scheduler

    Check your crontab and make sure there are no things in there that you didn’t put there yourself. If you can’t access your crontab or don’t know how to, ask your hosting provider for help.

44. Report the attack to your host

    Get your host to help you flush out the problems and make sure the attacker has not left any hidden backdoors to make it easy for him to regain access to your site.

45. Change your passwords

    Change passwords and, if you can, usernames to all access points to your site: MySQL databases, FTP users, administrators, SSH accounts, control panel logins, etc. Use strong passwords!

46. Restore a working backup, or reinstall completely

    Wipe your server clean and roll back to a known, safe backup. Make sure all security holes have been patched before you reopen the site. You should also consider deleting the entire installation and just reinstall Joomla and your extensions from scratch, so that you are certain not to include any backdoors a hacker may have left behind prior to the attack.

47. Report security issues you may find

    If you find soft spots in Joomla or any Joomla Extensions, report them on the security forum, but without outlining to the public how to hack one million Joomla sites in 10 seconds. Also contact extension developers directly to give them a chance to produce a patch for their users.

    When you report problems on the security forum, use the post assistant tool to make sure you include all the required information so that people are able to help you properly.

48. Official tips lists

    • Has your site been compromised? Read this.
    • Go through this Security Checklist
    • Read this before asking security support questions

Further reading, other resources

49. Read Nicholas’ blog

    Nicholas K. Dionysopoulos’ blog is currently, by far, the best source of new information on this subject.

50. Follow Jeff Channel (@jeffchannel)

    Jeff has found several security holes both in the Joomla core and in well-known Joomla extensions. He tweets regularly about security, and you could probably also hire him to do a security audit of your extension.

51. Joomla security forum

    Discuss at the Joomla 1.5 security forum.

52. Security checklist

    Joomla official security checklist

Acknowledgements

Thanks to Brian Teeman for valuable input and to Nicholas K. Dionysopoulos for his numerous articles on this subject!

The Joomla way, courtesy of corePHP.com

JLoader::import('joomla.application.component.model');
JLoader::import( 'items', JPATH_ADMINISTRATOR . DS . 'components' . DS . 'com_foo' . DS . 'models' );
$items_model = JModel::getInstance( Items', 'FooModel' );
$items_model->setState( 'id', $myItemId );
$items_model->get_item();

A detailed explanation of this code can be found in the original blog post. Basically, this code imports the core model class, then the component’s model and instantiates it. Then it sets a state and fetches an item. Easy enough. The point made by corePHP is that you can do this from anywhere, not just from one particular view, but also from say, modules or even other components.

The same MVC code, Nooku style

KFactory::get('admin::com.foo.model.items')
->id($myItemId)
->getItem();

Shorter yes, but not very different you say, somewhat disappointed, since you were expecting white rabbits to jump out of a hat as soon as “Nooku” was mentioned.

And where is that 80% source code reduction those Nooku developers always brag about?

If you just look at the character count, you will see that we have reduced the amount of code by 75% already. We coders like to talk lines though, so going from 5 to 3 lines looks like nothing more than a 40% reduction, right? Well… no, actually, there’s more to it! Lets look at the details:

• The Nooku Framework loads the classes you need, based on the identifier string “admin::com.foo.model.items”, so you don’t have to think about calling JLoader.
• Nooku has a fluent interface, and methods return the object so you can chain calls together. (notice only one semicolon)
• Notice: There is no id method (line 2) in the model. The call is therefore caught by the magic __call method in the framework model we’re using, which will act as a state setter for the model, and effectively it sets the id state to be $myItemId. Short & sexy.
• And now to the most interesting part: In a web application framework like Nooku, getting an item from a model based on an ID is perhaps simplest thing you can do. It’s only logical that the framework should be able to handle usecase out of the box. Therefore it’s very likely that in the Nooku version of this code, you wouldn’t find a FooModelItems class at all. What happens instead, is that when KFactory::get() is called and the class is not found, the framework falls back to a default class, which is more than capable of handling this usecase.

  So the really big code reduction here is that you don’t have to write the model class at all.

While I enjoyed corePHP’s tutorial on reusing a model and really agree with the concept of reusing code, it kind of just scratches the surface of what you can really do with very little code. So I am going to give a short intro to HMVC, with a practical Nooku example.

Moving on from MVC to HMVC

Yes, HMVC. That’s MVC with an H in front of it. HMVC looks like something more than just MVC too, doesn’t it? And it is! I won’t go into the details Martin Fowler style, but instead just give you a taste of what you can do with it.

Short explanation of HMVC: HMVC stands for Hierarchical Model View Controller. Normally, in MVC, you would have one controller setting states in one model, you’d be fetching data from the model and displaying them in a view. The three classes working together here (MVC) is called a triad. In HMVC though, you can have one triad call up and use another triad, so that multiple MVC triads are used together. You could even chain this further and have the second triad call up a third one.

Here’s a practical real-life example which I have lifted off the com_harbour extension written by Christian Hent (@christianhent):

In com_harbour, you have a port view, where information about a port is shown. This could be for example “The Port of New York”. Within the port view, you want to also display all the boats which are registered under this port.

Lets look at the code:

The basics: Showing the port info

I’ll not bore you with the details here. It’s a bit different than Joomla (read: easier), but you should be able to understand it:


<h1><?= $port->name; ?></h1>
<h2><?= @text('description'); ?></h2>
<p><?= $port->description; ?></p>
<h2><?= @text('boats'); ?></h2>
…


The HMVC part: Showing the port’s boats

Now, here I want to display the list of boats that hail to this port, and I accomplish that with these lines of code:

<?=
KFactory::tmp('site::com.harbour.controller.boat')
->harbour_port_id($port->id)
->layout('table')
->browse();
?>


That’s it!

Line–by-line explanation:

Line 1: <?= is a shorttag for <?php echo. This is parsed in Nooku and makes those handy shorttags compatible across webservers.
Line 2: Fetch the boat controller object, using KFactory::tmp() this time, so always creating a new object.
Line 3: Set harbour_port_id = $port->id
Line 4: Set layout = ‘table’
Line 5: Call the controller’s Browse action (Browse as the B in BREAD)

So what am I really doing here? Well, you could look at it as if I was displaying contents I had just fetched from this URL:
index.php?option=com_harbour&view=boats&layout=table&harbour_port_id=X

In other words: The boat controller is called, it will in turn set the harbour_port_id as a state in the model and register what layout to use. Then, when the browse action is called (Observe: Browse = show multiple items, Read = show one item), the model will return all items that matches it’s current state and they will be displayed in the table layout.

…and all of that happens with one line of code, in our ports view.

This is just one of the basic ways that Nooku Framework makes life as a Joomla developer worth living again :)

Read more on HMVC

Split views using Nooku Framework
http://www.codewalkers.com/c/a/Miscellaneous-Code/A-Sample-Web-Application-with-the-HMVC-Design-Pattern/
http://net.tutsplus.com/tutorials/php/hvmc-an-introduction-and-application/

The frameworks and the CCKs

The Joomla Framework and the Joomla content component are two of the most vital parts of Joomla if you ask me. Now, look at the number of extensions that exists that are replacing these core parts in improved ways:

K2, jSeblod, FLEXIcontent, Sobi2 and others are dominating in the CCK/ACL section of things, improving content management in Joomla greatly. Gantry, Morph and T3 are template frameworks looking to make template production easier and templates more stable and feature rich. And for the core framework itself there are multiple replacements available, like Nooku, which is Joomla flavoured and my favourite. Others again, like JomSocial, are just to replacing the Joomla core framework with something entirely different.

So what am I really saying here?

This shows that Joomla is not providing the functionality that the community needs. It is not able to keep up with the pace of which the world is moving, so people go elsewhere to get their fix, or they create their own solutions to scratch their own itch. Joomla is, as Hannes Papenberg points out, already years behind in development, and while Joomla 1.6 may and may not be too far away, it is still outgunned by solutions already available to us. Even by Joomla extensions.

So, development is too slow. What can we do?

A clear choice, as suggested by Daniel Chapman and others, is for Joomla to pick up existing best-of-breed GPL frameworks that are Joomla compatible, and integrate them into the core, to make up for lost time.

This would bring J! up to date almost overnight, and reduce the confusion and bloat of having different fw’s for every major extension you install.
- Daniel Chapman

Don’t get me wrong: I’m a supporter of the idea of a slim and light core. Today’s core is packed with outdated extensions that have passed their expiration date a long time ago. When the framework changes, very often all the components have to be changed too. I hardly ever even use them personally. They have become a burden, and should be removed:

The content component is poor on features compared to most of the CCKs, and don’t get me started on com_contacts or com_weblinks. Com_search does an awful job at searching and does not prioritise the search hits at all. Routing is a pain, even for seasoned developers. Com_mailto, com_newsfeeds, com_contact, com_banners… They all have extensions that does the job better.

I’m not saying we should pack Joomla with cool new extensions, just that we need a solid foundation to build on and make it easier to create cool, quality extensions. And that’s where frameworks come in handy.

So why won’t Joomla adapt external frameworks?

Volunteer contributions are worthless

I am no Deep Throat insider, but in my opinion, the decision to give a salary to two core developers and not any other contributors at all, was a landmark in Joomla history. With this, Joomla effectively put a price tag on the projects estimated value of volunteer community contributions, and the value was $0. If I had still been a working group member and contributor, and then suddenly seeing Open Source Matters decide that two people should be paid for their work while the rest of us were deemed unworthy, I would not be very happy.

Paying developers like this can also get in the way of the natural flow of contributors in FOSS projects. A paid developer suddenly has lots of reasons to cling on to his job and shut other contributors out. Instead, lead developers should be encouraged to open things up, or even pass on their responsibility at some point to someone with new ideas and loads of fresh motivation. Money just may be too good of a motivation not to do so.

I think this is part of the cause of Joomla’s low bus factor. The decreasing community contributions is a problem that is beginning to sink in with the leadership too. Strength may not necessarily be in numbers, but a lack of numbers is taking it’s toll.

Community is taking charge

This and other things, for instance the Joomla leadership’s deliberate ignoring of the first community organized international Joomla event, J and Beyond 2010, is truly a sad turn of events for the project itself. Not only was the JAB10 event ignored, it was also officially and directly counteracted, by joomla.org promoting other events instead.

If someone decided to take it upon themselves to host an event to promote my product, and pay for it themselves too, I would make sure that they got as much attention and credit for their work as I could possibly give them.

On the bright side for Joomla though, community is stepping in where leadership is having issues. Like with ATAAW, JAB10 (see also #jab11!), JFWD, JCM and J!UX, consisting of volunteers aiming to contribute to Joomla in various ways.

Your turn

If you made it this far, you’re obviously interested in Joomla, and probably using it already. What are your hopes for the future? Do you thing there is still hope for even the most zealous Joomla fans out there? In which direction would you like the project to go?

Design patterns

Design patterns are reusable solutions

First of all, MVC is an architectural design pattern. Yeah, that’s a mouthful. What is a design pattern? Well think of it like this: Patterns are stuff that repeats itself, right? So a design pattern is a design that repeats itself, or you could say “is reused”. A design can in development terms be thought of as a solution, so when you hear the words “design pattern”, you can think reusable solution.

Reusable solutions are smart ways of solving similar problems without reinventing the wheel. An architectural design pattern like MVC is in other words a reusable solution to an architectural problem. Many computer softwares aren’t that different architecturally, so MVC is a solution that can be used in many circumstances.

By giving names to these design patterns, for instance MVC, developers can easily communicate between one another how they are approaching or solving issues in their applications. For instance, if a PHP class in my code is called ArticleController (where “Controller” is the C in MVC), another developer that reads this code will immediately understand what the responsibility of this class is and what it should be able to do, because he understands the responsibility of the Controller in the MVC pattern.

…like the wheel

Speaking of not reinventing the wheel; the wheel itself could be seen as a design pattern:

We use the wheel in for instance cars, bicycles, buses and trains. So the wheel has indeed become a reusable solution to one common challenge: Making a vehicle easy to move. The wheels themselves are different on each vehicle because they are adapted to suit special circumstances, but the wheel concept itself is the same: Once the solution has been identified, it’s just a matter of putting it to good use.

So if you and I are building a vehicle that’s going to solve the challenge of transporting us from A to B, and I suggest that “we should put wheels on this thing”, then you’ll basically know what I’m thinking and we’ll also have a common understanding of how we’re going to build it. If we were to travel on a snow covered surface though, you could have suggested that we used a pair of skis as our solution, instead of the wheels. I would of course have slapped my forehead and agreed with you, without needing to see complicated blueprints or having long committee meetings.

That’s the advantage of having design patterns!

Jeff Atwood also has another nice MVC analogy in a blog post of his, see the link in the references & footnotes at the bottom.

The building blocks of MVC

The MVC pattern splits your code into three parts, often referred to as a triad: The Model, View and Controller. Here is a brief outline of what job each part does, particularly in the Joomla and Nooku Frameworks.

Note: This will teach you how to talk the talk, but not walk the walk! Oleg Nesterov’s presentation “Joomla Extension Kung Fu” provides some nice PHP code examples you can look at though to see MVC in action. See the link in the references & footnotes at the bottom of this article. I’d also recommend looking into the Nooku Framework sourcecode to see how MVC is implemented in PHP there.

Here are the basics though:

Controller

The controller is in charge, as the name would suggest. When you do a request, like for instance in a Joomla site entering example.com/index.php?option=com_content&view=article&id=13 in your browser, the controller in the content component takes care of getting the request data. For this request, we have the variables option, view and id visible in the URL. From that, the controller will set the model state so that the model, which is fetching the data for us later on, will know that it needs to fetch article number 13. The controller will then tell the article view to render itself.

View

The view decides what ends up on the screen. It will use the model to fetch the data it needs, load the template and display the content inside that template. In this case, the view requested is “article”, singular. In Nooku Framework, the view will automatically deduct from this that it needs to fetch a single item instead of a collection of items, as “articles” (plural) would have suggested. This is a small part of the “magic” happening underneath the hood in Nooku, which makes it so easy to work with.

Model

The model handles the data. If you need something added, edited, deleted or fetched to or from the database, the model is the class that will do the heavy lifting. The controller will set it’s states (like filtering, ordering etc), and then the model will do whatever it is told to do.

Benefits and advantages of MVC, for all of us

Why should you care about MVC and demand that your developers do the same?

A clear separation of the code

First of all, MVC splits your code into three distinct parts, each of which have clear responsibilities and limitations. When you’ve used this pattern for a while, deciding where to put a piece of code or look for a piece of code becomes intuitive and straight forward. This makes many parts of development a lot easier, because the design pattern dictates where the code should be located and what the code should do.
It also makes it easier for other developers who know MVC to read and understand the code you have written, and after that, also extend or contribute to it, or even replace parts of it.

Visual flexibility

At the heart of MVC there is the desire to separate the presentation of data from the manipulation of data, partially because we want to reuse as much code as possible, but also because we want to be able to present one set of data in multiple ways. In the Joomla CMS for instance (version 1.5 and up), you have templates, and you can change templates to easily get a list of articles to look completely different without modifying the Joomla core code. Each template has the power to implement it’s own overrides for any view in any component in Joomla. Consequently, anyone can use their own template to make a component look exactly like they want to, given that the component in question follows the MVC design pattern of course.

Formatting flexibility

A new presentation of content need not only mean a new HTML template though: A list of articles can very well be displayed in a completely different format, like CSV, RSS, PDF or JSON. This can allow us to easily be able to reuse it in other parts of your website or even on other websites. So while the presentation or format of the data is different, the functionality used to fetch the data from the database is exactly the same every time.

Replaceable parts

On the opposite side: If I suddenly decide I want to store my articles in flat files instead of in a database, I can just modify the part of MVC that stores and fetches data, and leave the presentation part alone. The presentation of the data is in other words decoupled from the part of the application that communicates with the storage of the data.

Workflow

My personal experience is that when working in teams, the separation of responsibilities in MVC makes it easy to divide the workload between team members. If a task on my task list just says “Article model”, then these two words also describe pretty well what it is that I need to do.

Where to go from here?

So how do I find out if my favourite component X follows the MVC pattern or not? Easy in most systems! In Joomla you can open the component’s folders, either administrator/components/com_x/ or components/com_x/ in a file browser or FTP client, and look for folders named “models” and “views”. If you find none, then the component is likely not using MVC the Joomla way.

If so, make sure to send the component’s developer an e-mail with a link to this article ;)

Conclusions

Design patterns are like wheels: They make things roll smoothly!

The benefits of the MVC design pattern: It provides a logical and predictable architecture to work with and collaborate on, it allows us to reuse larger parts of our code, and it allows us to modify parts of our code while other parts can remain untouched. Less code equals less maintenance, and more decoupling of code gives better flexibility and extensibility.

More importantly: MVC is the Joomla standard, so if you’re building extensions for Joomla you should use MVC like the rest of us.

References & further reading

1. GUI Architectures, by Martin Fowler
2. Joomla Extension Kung Fu, presentation by Oleg Nesterov given at J and Beyond 2010
3. Design patterns illustrated, presentation by Herman Peeren given at J and Beyond 2010
4. Understanding Model-View-Controller, by Jeff Atwood
5. The M in MVC: Why Models are Misunderstood and Unappreciated, by Pádraic Brady
6. Benefits of the MVC design pattern, by the IBM Corporation
7. Wikipedia article on MVC, which clearly states that MVC was coined by a Norwegian guy, so you’re obviously in good hands here! :)
8. Patterns for beginners, by Falk Bruegmann

Preface

No, not “preface” as in “this is the start of a long and boring novel”. It’s just two quick preliminary pieces of information that you need to be aware of:

1. Nooku Framework is not to be confused with Nooku Content. Nooku Content is more a competitor of Joom!Fish: It’s a Joomla component used for translating content. Nooku Content is also distributed only among paying partners in a partner programme. Nooku Framework however, is a programming framework, and it’s free and open source software (FOSS). Both products are backed by the same developers though, hence the similar names.
2. When I say framework, I really mean software framework, a term which Wikipedia defines and explains really nicely. But don’t click that Wikipedia link for now. Instead, let me give you the short story, from a Joomla perspective:

What is a framework?

A framework allows you to reuse common programming code, which is a good thing. Let me explain why by comparing a framework to something a lot easier: Your e-mail signature. In your e-mail software, like Outlook or Apple Mail, you write your e-mail signature once in your settings, then automatically use it whenever you send an e-mail.

This means:

• You spend less time writing your e-mail.
• Time saved means you have more time to do what’s important: Focusing on the contents of your e-mail.
• You get the signature right each time; In a stressful situation you might forget to include your phone number or even type your name incorrectly, but not when you’re using a predefined signature.

The same thing goes for software frameworks from a programmer’s perspective:

• It saves you time by eliminating the need to do repetitive tasks, which means you can focus on what’s important: Finishing the application and making sure it meets the requirements.
• You get less bugs in your system, because the parts handled by the framework have been used and tested earlier, and are usually bug free. So the more framework code you use, the less bugs you should experience. More time saved.
• An added bonus: A framework with multiple users allows people to find bugs together, in their shared, common code.

So what’s the difference between Nooku, Joomla and the Gantry Frameworks?

I see many people asking this question, so I just had to answer it once and for all:

Nooku and Joomla try to accomplish the same thing: To make developing Joomla components, plugins and modules easier. This is primarily stuff happening under the hood you might say. Gantry is a framework made by Rockettheme to make developing templates easier, which is more the facade of your website.

In other words, you can’t compare Gantry to the other two. Instead you should compare Gantry to the Morph Framework, which is rumoured to be Nooku based in the future, but that’s a whole different story.

Why do I prefer Nooku Framework over Joomla’s own framework then?

I have been a part of the Mambo/Joomla community since 2004, and I have written components both using the Joomla Framework and the Nooku Framework. So I have tried and tested both, and have decided to use Nooku for the following reasons:

• It’s better. For me as a programmer. It reduces the amount of code I need to write by something close to 80%¹, which means more time for me to spend on creating new and cool stuff, and less bugs for you to get annoyed over. The only two programmers that I know of that have looked into Nooku and still decided they preferred the Joomla Framework, have been OSM employees. I.e. the guys who are paid to develop the Joomla Framework.
• It makes me better. Nooku reuses solutions to common problems (called using design patterns) to a greater extent, and unlike Joomla it forces me to adhere to conventions and standards. No slacking around in other words. The predictability in the coding combined with strict data filtering gives my components easier extensibility and my clients better security. I’ve recently also started using HMVC, and I’m loving it.
• It’s Joomla compatible. Any Joomla website that needs custom functionality can start using Nooku to build that functionality, today. It installs like any other extension, and does not interfere with what’s already on your Joomla site and using the Joomla Framework.
• It’s cheaper. I know; too obvious. Less time spent programming + Less bugs = Lower development and maintenance costs.
• It’s the second generation. Yes, literally the second generation. You may now know, but both the Joomla and Nooku Frameworks were built basically by the same guy: Johan Janssens. In fact, Johan is still considered to be the largest contributor to the Joomla codebase (by far!), even though he stopped working with Joomla in January 2008. He took with him the knowledge and lessons learned, and started from scratch to build something better: Nooku.
• It’s got a developer community. Unlike Joomla, development at the framework level is moving forward at a rapid pace, there is lots of activity, and developers are sharing and contributing code between themselves. In the web development industry, things are moving forward very quickly, and for the sake of myself and my clients, I need to keep up and stay sharp.

So I choose Nooku.

If you’re a programmer yourself, go check it out at http://www.nooku.org/framework.html. If not, you should talk to whoever does the programming for you and make sure that they do.

Footnotes

1. The 80% figure is borrowed from NinjaBoard’s comparison. This, though, dates back to October 2009, after which the Nooku Framework has improved further, so the number should probably be higher than 80%.

Improvement suggestions

I have listed a few of these below, and please add comments to this post if you find some of your own or you disagree with me, and I’ll adjust this list accordingly.

Naming conventions

I primarily want to address the lack of standards in this schema, which I even reported back as early as in 2006 when creating the 1.5 schema. In coding, the concept of convention over configuration gives predictability, simplicity and reduces the amount code you have to write later on. Same goes for database schemas and Joomla: The better the schemas follow the naming conventions, the less PHP code you’ll have to write later on. Convention also eliminates ambiguity, and developers won’t be tempted into guessing. Just having a convention for column names in the database would be a nice first step, both for Joomla and 3rd party developers alike. Right now, column names for instance are a sweet mix of abbreviations and non-abbreviations, both underscored and camelCased names. Plus, the column id is used to name most INT primary key names but not all.

A prime example of Joomla’s lack of conventions can be seen in the content table, where we have these five foreign key columns: catid, sectionid, language, checked_out and asset_id. The first is abbreviated with no underscore, the second is also without underscore but not abbreviated, the third does not look like a foreign key because it has no “id” in it’s name like the others, but it holds the language code for the article, and thus references language.lang_code (Who would have known?). The fourth contains has no “id”-hint, and no hint to the fact that it refers to users.id. Finally, the fifth column is not abbreviated but with underscores (the way I like it personally). Five different naming conventions in one single table, if we should at all should acknowledge “checked_out” as some sort of readable standard foreign key for user.id.

Also check out and compare content.created_by and categories.created_user_id. They both reference the same column. Weird huh? See the lack of convention? Which format do you prefer?

Table names

For consistency, I would propose changing using plural names on tables. Just a little something I got used to while using Nooku. Most tables are already pluralised, and tables normally contain multiple rows, so it makes most sense to use pluralisation as the standard and thus renaming the following tables:

1. content renamed to contents
2. content_frontpage renamed to contents_frontpage
3. content_rating renamed to contents_ratings
4. menu renamed to menus
5. modules_menu renamed to modules_menus
6. session renamed to sessions

Primary keys

Primary keys are important, as they are used for row identification, and will therefore play a big part in the PHP code we write for our applications and extensions.

Names

Most tables have a primary key column named “id”, but not all tables follow this rule. The languages table has lang_id, update_sites has update_site_id, update_categories has categoryid, etc. In other words: Lots of inconsistensies and no clear naming convention.

Nooku is a good example of a framework that imposes a convention which allows the PHP-code to interpret the database structure automatically, and only requires you to write code for those few special cases that don’t fit the blueprint at all. Nooku uses the following convention for primary key names: componentname_tablename_id. For the articles table in com_content, the Nooku table name would have been content_articles, and the primary key would have been content_article_id. Notice the plural formed table name and the singular column name. Logical, since the table represents a collection of items, and the column identifies a single one.

Next, I would wish for the foreign keys to also use the full name of the column they are referencing. So instead of those abbreviated catid columns all over the place, you use for instance category_id.

Field types

A lot of the tables have an auto increment INT type of primary key, which is to be expected, but there is no clear standard as to what datatype this field should be. Some are unsigned, some are not, some are INT, some are INT(10) and some INT(11). There is no reason why these should not just follow a common standard.

The INT key sickness

Very often you will find yourself using a primary key column for your table named something_id, and it will have the field type in the shape of a large integer. This is typical for relational databases, so don’t be alarmed, but don’t stop thinking either. The primary key in a table is something that can uniquely identify each and every record in that table. If you already have such a column, then there is no need to add another integer.

Consider these examples from the Joomla database:

The language table has a lang_id INT(11) column. This really seems redundant, as the lang_code field, CHAR(7), uniquely identifies the language just fine. In the menu table, and all other tables using multilingual features, you have the column language which is also a CHAR(7), and references lang_code. It’s weird, and bad code, to use a foreign key to reference a column that’s not a primary key, and not even unique. It’s also hard to see the connection from menu.language to languages.lang_code too. As before: Consistent naming would really help; why not call it menu.lang_code instead?

Similarly, in the redirect links table, the old_url-field is what really should identify redirect links. You can’t have multiple redirects for one old url, hence every row in this table is uniquely identified by it’s old_url column, so that should be the table redirect_links’ primary key. Consequently, the id INT field seems redundant here too.

An oddity: ACL

Viewlevels are in the database connected to usergroups via JSON-encoded data in the column viewlevel.rules. The column name is misleading, but it is even more strange to see foreign keys stacked up in a pure JSON-encoded field like this, instead of for instance creating a table for usergroups_viewlevels, to connect the usergroups and viewlevels tables, similar to for instance modules_menu, which connect modules to menu items.

There is probably an explenation here, and it probably got something to do with performance, pragmatism or laziness. I just think it’s bad database design, that’s all. Maybe someone out there, who has been deep into the ACL system, can explain this to me?

Also strange: Extension update sites

This part of the database seems incomplete and unfinished, even though this schema is supposed to be a beta. For instance the update_sites_extensions table allows null values in it’s two fields extension_id and update_site_id, and a unique key is nowhere to be found in the table. This might just not be a feature due for Joomla 1.6, but then I would not see the reason for introducing these tables in the first place.

Other things

There are so many other things, but I can’t be bothered to write them all down. I actually started writing a section in this article about recommendations for normalization, changes in column names, types and usages, but the list already had six elements before I was finished going through two of the ten or twelve large tables in this schema.

This database schema suffers because of legacy issues, and a parts of it probably dates back even as far as the Mambo days. You can really see the same things happening in the user interface. Or should I say not happening? Instead of being a constantly renewed, improved and refactored CMS and framework, Joomla bears both the visual and codewise smell of patchwork and old legacy code.

Because of the obvious lack of progress in the Joomla project, I fear that this will be the last database schema for Joomla I will make; The web world is moving forward, but Joomla seems to be stuck. We strive to provide our clients and ourselves with the best tools of the trade, and that’s what I’ll keep doing. I hope I am proven wrong.

Update: With the release of 1.6 beta 5, I just did a diff between the schema for 1.6 beta 2 and 1.6 beta 5, and they are identical.

DOWNLOAD available in different formats:

• MySQL Workbench format (.mwb), 90 kB needs MySQL Workbench to use.
• PNG format, 2652 * 2658 pixels, 800 kB
• PDF format, 1 MB
• SVG format, 2 MB, to view directly in modern browsers

Schema modifications

Here are some modifications that I personally have done in various places in the schema to be able to reproduce it visually in MySQL Workbench, and make it more readable. This does mean that that this schema is not 100% identical to an actual Joomla 1.6 installation, because of these changes. For instance it also has foreign key relations drawn in, whereas MySQL’s MyISAM tables don’t support those. This schema should therefore only be used as a visual supplement to Joomla development.

Here are the changes that makes this schema differ from an actual Joomla 1.6 installation:

Invisible relations

This is not something I have changed, but something I deliberately did not touch. For claritys sake, some foreign relations to the users table have been left out. For instance created_by, modified_by, locked_by, created_user_id, modified_user_id and checked_out all reference the field users.id. If I was to set up all these relations, the schema would become so crowded with relationship indicators that it would be virtually unreadable.

There is also a “relation” here between the tables viewlevels and usergroups, but that’s not solved in SQL: All usergroups that belong to a viewlevel are put into an array that is JSON encoded and stored in the viewlevel.rules field. This is not a foreign key in SQL, so the relation is not show in the schema.

Changed column types

If a column is referencing another column in the database (like a foreign key), these columns need to be the same datatype. A lot of the time, this is not the case in Joomla, for some strange reason. To represent real foreign key relations in MySQL Workbench, this needed to be changed, or else MWB would not even allow the relations to be made at all. We have had issues like these all the way back since Joomla 1.0. Hopefully they are picked up and fixed soon, because fixing this should not be a problem for backwards compatibility.

Here are the columns I have had to change:

1. modules_menu.moduleid has been changed from an INT to an INT(11), since it references modules.id
2. In two places, asset_id foreign key was of datatype INT, but referencing asset.id it should have been INT(10), so those foreign key relations have changed, and this affects the following tables; categories, content
3. Similar to asset_id, catid columns were also of datatype INT, whereas they should have been INT(11) since they are referring to categories.id, which is an INT(11). This occured in the following tables: banners, contact_details, content, newsfeeds, weblinks
4. user_usergroup_map.user_id, messages_cfg.user_id, messages.user_id_from, messages_user_id_to all reference user.id, hence they can’t be an unsigned INT, but has to be a signed INT.
5. Multiple occurences of user_id references user.id, so they must be INT, not INT(11). This affects the following tables: user_profiles, session (although the field is named userid in the session table, without the underscore).
6. schemas.extension_id has been changed from INT(11) to INT, since it references extensions.extension_id
7. update_sites_extensions.update_site_id changed from INT to INT(11), since it references update_sites.update_site_id, which is an INT(11). Both columns in update_sites_extensions got a NOT NULL attribute added, since they make no sense without it.

Changed column ordering

Column ordering in some tables has been changed so that foreign keys are listed just below primary keys, as I feel that this is cleaner and makes the relations easier to read. The changes were so many that I gave up trying to write a complete list.

All in all, this schema badly needs an overhaul, to say it mildly. I have written a short blog post explaining why I think the Joomla database schema smells.

These are just very brief instructions, specifically on how I did this on my own system. A complete documentation on Phing can be found on at phing.info. You don’t actually need MAMP either, it was just what I was working with to begin with.

Open the application Terminal. You can find pear by doing a locate call: locate pear.
Currently MAMP requires you to install it into /Applications, so you should find pear (for php5) here: /Applications/MAMP/bin/php5/bin/.
Make sure MAMP is using the correct php version, by starting MAMP and checking your preferences.

1: Go to the correct folder:
cd /Applications/MAMP/bin/php5/bin/

2: Make sure pear can locate the package:
./pear channel-discover pear.phing.info

3: Install phing:
./pear install phing/phing

Phing should now be installed. Run the command “ls -al” and you should see phing in the same folder as pear.

You can now run this command:
./phing -h
This will give you an overview of the phing’s options.

From here you can for instance just run:
./phing -f /path/to/your/build.xml