Now you need to create and distribute an update script with the next version of your component, and MySQL Workbench makes creating this update script really easy. Here is what you need to do:

Install MySQL Workbench

I’ll not help you with this. Just go to the MySQL Workbench homepage and follow the instructions from there. Multiple platforms supported! It’s the same tool I used to create the Joomla 1.6 database schema.

After installation, start MySQL Workbench and we’re off!

Create a new model

From the main menu, click File -> New Model, then double click Add Diagram. Save your diagram right away and give your file a name. It always sucks to lose your work.

Basic settings

First some settings that are handy:

To the right, under the tab “catalog”, you’ll find your schema with the default name “mydb”. The schema name is the database name by the way. Double click it and change it’s default collation to utf8 – utf8_general_ci. UTF8 is what Joomla uses by default.

Open MySQL Workbench Preferences and set the column default names and values for the Model configuration. That way you can make it easier for yourself to follow the standards and conventions you want to be following. For those of you who are using Nooku, make sure to look up the Nooku naming conventions.

For my example I’ll be setting up a small blog database.

The articles table

Click the Place a New Table-button in the left toolbar and click anywhere on the canvas to place a new table. Double click the table to start editing. You’ll see a line of tabs appearing at the bottom of the screen, one of which being columns, which you can edit to add columns to your table.

I created an example table structure for a blog articles table that you can look at on the screenshot to the right. Very basic stuff, just an id, a title, some content and a publish date for every article.

Comments for my blog articles

I then added a comments table, so that people can add comments to my blog. Basic stuff here too: Comment id, author name, author email, author homepage and comment contents. Note: I added no relation between the two tables just yet.

Now, I need to set up a relation between the two tables, since each comment should belong to a blog article. I click the button for 1:n non-identifying relationship, because each blog post will have potentially many comments attached to it. Non-identifying just means that the relation does not identify the comments, in this case that’s the job of the comment id.

So after clicking 1:n non identifying, I then click the comments table and proceed to click the articles table to set up the relation. You’ll see the line representing the relationship popping up immediately. A new column, blog_article_id is also added to the comments table automatically. This is of course the foreign key.

I then click and drag the new column, blog_article_id, up in the comments table, so that it’s ordered as the second column. Not necessary, but I like to keep my keys close. Note that you can also double click the relation indicator itself to edit the relation and manually change it or indicate if it’s mandatory or not. A comment needs to belong to a blog post, so in my example the relation is mandatory.

There, all ready to go!

Create the components install SQL script

First, I forgot, my tables need to be able to have their names converted to use any db prefix according to the Joomla standards, so I need them to have the db prefix nameholder #__. So blog_articles becomes #__blog_articles and blog_comments becomes #__blog_comments.

Ok, now let’s create an install script. Go to the main menu and click File -> Export -> Forward engeneer SQL CREATE script. For Output SQL scriptfile, put in a the desired filename for your SQL file, install.sql would fit the bill. Select “Omit schema qualifier in object names”, or else the database name (mydb) will be included in all SQL statements in the script. You can also skip creation of foreign keys if you want to since they have no effect in MyISAM, but it will do no harm to leave them in there. In my opinion they just makes the SQL structure easier to read for people interested in checking out it’s details.

In the next step, choose to export table objects. You can also add default data to your tables if you want to, and would then wish to also export data for the tables. Click continue, then finish, and the SQL file will be created for you.

Creating version 2 of my component, and the upgrade SQL script

Then I wanted to add an author name to my blog articles, so I went ahead and did that. Now I have a new version of my database, and I need to update my component. So my components version two will have to include an update script. For this I’ll need some SQL to modify the table structure, and this is where MySQL Workbench comes in very handy.

After having modified the schema, from the main menu click File -> Export -> Forward engineer SQL ALTER script. In the Input file-field, select the SQL CREATE script you created earlier, and in Output file you name the resulting file. This will take the SQL file you just created (version 1), compare it to the current updated schema (version 2) which includes the author_name, and then produce an SQL ALTER script that creates a bridge between versions 1 and 2. that you can run on existing installations to update their schemas. Pretty neat, huh? Takes five seconds, and it’s easy to include in your upcoming new product release!

This example might not be much, but imagine having done multiple small and big changes. Having easy version creation and automation of upgrade scripts at hand then really comes in handy.

Streamlining the process

All we need now is some creative soul to write some Lua scripts to automate the creation of update scripts for all old versions every time a new version is saved. When you create version 20 and run the plugin, then all update scripts for all previous 19 versions are automatically set up for you.

Automatic insertion of $Id$ tag for SVN keywords would also be nice of course. Validation of Joomla compatibility or Nooku conformity are other ideas. Anything to prevent potential errors and automate tedious tasks really.

There exists a plugin for doing Doctrine exports from MWB, so someone might get some inspiration from that.

Improvement suggestions

I have listed a few of these below, and please add comments to this post if you find some of your own or you disagree with me, and I’ll adjust this list accordingly.

Naming conventions

I primarily want to address the lack of standards in this schema, which I even reported back as early as in 2006 when creating the 1.5 schema. In coding, the concept of convention over configuration gives predictability, simplicity and reduces the amount code you have to write later on. Same goes for database schemas and Joomla: The better the schemas follow the naming conventions, the less PHP code you’ll have to write later on. Convention also eliminates ambiguity, and developers won’t be tempted into guessing. Just having a convention for column names in the database would be a nice first step, both for Joomla and 3rd party developers alike. Right now, column names for instance are a sweet mix of abbreviations and non-abbreviations, both underscored and camelCased names. Plus, the column id is used to name most INT primary key names but not all.

A prime example of Joomla’s lack of conventions can be seen in the content table, where we have these five foreign key columns: catid, sectionid, language, checked_out and asset_id. The first is abbreviated with no underscore, the second is also without underscore but not abbreviated, the third does not look like a foreign key because it has no “id” in it’s name like the others, but it holds the language code for the article, and thus references language.lang_code (Who would have known?). The fourth contains has no “id”-hint, and no hint to the fact that it refers to users.id. Finally, the fifth column is not abbreviated but with underscores (the way I like it personally). Five different naming conventions in one single table, if we should at all should acknowledge “checked_out” as some sort of readable standard foreign key for user.id.

Also check out and compare content.created_by and categories.created_user_id. They both reference the same column. Weird huh? See the lack of convention? Which format do you prefer?

Table names

For consistency, I would propose changing using plural names on tables. Just a little something I got used to while using Nooku. Most tables are already pluralised, and tables normally contain multiple rows, so it makes most sense to use pluralisation as the standard and thus renaming the following tables:

1. content renamed to contents
2. content_frontpage renamed to contents_frontpage
3. content_rating renamed to contents_ratings
4. menu renamed to menus
5. modules_menu renamed to modules_menus
6. session renamed to sessions

Primary keys

Primary keys are important, as they are used for row identification, and will therefore play a big part in the PHP code we write for our applications and extensions.

Names

Most tables have a primary key column named “id”, but not all tables follow this rule. The languages table has lang_id, update_sites has update_site_id, update_categories has categoryid, etc. In other words: Lots of inconsistensies and no clear naming convention.

Nooku is a good example of a framework that imposes a convention which allows the PHP-code to interpret the database structure automatically, and only requires you to write code for those few special cases that don’t fit the blueprint at all. Nooku uses the following convention for primary key names: componentname_tablename_id. For the articles table in com_content, the Nooku table name would have been content_articles, and the primary key would have been content_article_id. Notice the plural formed table name and the singular column name. Logical, since the table represents a collection of items, and the column identifies a single one.

Next, I would wish for the foreign keys to also use the full name of the column they are referencing. So instead of those abbreviated catid columns all over the place, you use for instance category_id.

Field types

A lot of the tables have an auto increment INT type of primary key, which is to be expected, but there is no clear standard as to what datatype this field should be. Some are unsigned, some are not, some are INT, some are INT(10) and some INT(11). There is no reason why these should not just follow a common standard.

The INT key sickness

Very often you will find yourself using a primary key column for your table named something_id, and it will have the field type in the shape of a large integer. This is typical for relational databases, so don’t be alarmed, but don’t stop thinking either. The primary key in a table is something that can uniquely identify each and every record in that table. If you already have such a column, then there is no need to add another integer.

Consider these examples from the Joomla database:

The language table has a lang_id INT(11) column. This really seems redundant, as the lang_code field, CHAR(7), uniquely identifies the language just fine. In the menu table, and all other tables using multilingual features, you have the column language which is also a CHAR(7), and references lang_code. It’s weird, and bad code, to use a foreign key to reference a column that’s not a primary key, and not even unique. It’s also hard to see the connection from menu.language to languages.lang_code too. As before: Consistent naming would really help; why not call it menu.lang_code instead?

Similarly, in the redirect links table, the old_url-field is what really should identify redirect links. You can’t have multiple redirects for one old url, hence every row in this table is uniquely identified by it’s old_url column, so that should be the table redirect_links’ primary key. Consequently, the id INT field seems redundant here too.

An oddity: ACL

Viewlevels are in the database connected to usergroups via JSON-encoded data in the column viewlevel.rules. The column name is misleading, but it is even more strange to see foreign keys stacked up in a pure JSON-encoded field like this, instead of for instance creating a table for usergroups_viewlevels, to connect the usergroups and viewlevels tables, similar to for instance modules_menu, which connect modules to menu items.

There is probably an explenation here, and it probably got something to do with performance, pragmatism or laziness. I just think it’s bad database design, that’s all. Maybe someone out there, who has been deep into the ACL system, can explain this to me?

Also strange: Extension update sites

This part of the database seems incomplete and unfinished, even though this schema is supposed to be a beta. For instance the update_sites_extensions table allows null values in it’s two fields extension_id and update_site_id, and a unique key is nowhere to be found in the table. This might just not be a feature due for Joomla 1.6, but then I would not see the reason for introducing these tables in the first place.

Other things

There are so many other things, but I can’t be bothered to write them all down. I actually started writing a section in this article about recommendations for normalization, changes in column names, types and usages, but the list already had six elements before I was finished going through two of the ten or twelve large tables in this schema.

This database schema suffers because of legacy issues, and a parts of it probably dates back even as far as the Mambo days. You can really see the same things happening in the user interface. Or should I say not happening? Instead of being a constantly renewed, improved and refactored CMS and framework, Joomla bears both the visual and codewise smell of patchwork and old legacy code.

Because of the obvious lack of progress in the Joomla project, I fear that this will be the last database schema for Joomla I will make; The web world is moving forward, but Joomla seems to be stuck. We strive to provide our clients and ourselves with the best tools of the trade, and that’s what I’ll keep doing. I hope I am proven wrong.

Update: With the release of 1.6 beta 5, I just did a diff between the schema for 1.6 beta 2 and 1.6 beta 5, and they are identical.

DOWNLOAD available in different formats:

• MySQL Workbench format (.mwb), 90 kB needs MySQL Workbench to use.
• PNG format, 2652 * 2658 pixels, 800 kB
• PDF format, 1 MB
• SVG format, 2 MB, to view directly in modern browsers

Schema modifications

Here are some modifications that I personally have done in various places in the schema to be able to reproduce it visually in MySQL Workbench, and make it more readable. This does mean that that this schema is not 100% identical to an actual Joomla 1.6 installation, because of these changes. For instance it also has foreign key relations drawn in, whereas MySQL’s MyISAM tables don’t support those. This schema should therefore only be used as a visual supplement to Joomla development.

Here are the changes that makes this schema differ from an actual Joomla 1.6 installation:

Invisible relations

This is not something I have changed, but something I deliberately did not touch. For claritys sake, some foreign relations to the users table have been left out. For instance created_by, modified_by, locked_by, created_user_id, modified_user_id and checked_out all reference the field users.id. If I was to set up all these relations, the schema would become so crowded with relationship indicators that it would be virtually unreadable.

There is also a “relation” here between the tables viewlevels and usergroups, but that’s not solved in SQL: All usergroups that belong to a viewlevel are put into an array that is JSON encoded and stored in the viewlevel.rules field. This is not a foreign key in SQL, so the relation is not show in the schema.

Changed column types

If a column is referencing another column in the database (like a foreign key), these columns need to be the same datatype. A lot of the time, this is not the case in Joomla, for some strange reason. To represent real foreign key relations in MySQL Workbench, this needed to be changed, or else MWB would not even allow the relations to be made at all. We have had issues like these all the way back since Joomla 1.0. Hopefully they are picked up and fixed soon, because fixing this should not be a problem for backwards compatibility.

Here are the columns I have had to change:

1. modules_menu.moduleid has been changed from an INT to an INT(11), since it references modules.id
2. In two places, asset_id foreign key was of datatype INT, but referencing asset.id it should have been INT(10), so those foreign key relations have changed, and this affects the following tables; categories, content
3. Similar to asset_id, catid columns were also of datatype INT, whereas they should have been INT(11) since they are referring to categories.id, which is an INT(11). This occured in the following tables: banners, contact_details, content, newsfeeds, weblinks
4. user_usergroup_map.user_id, messages_cfg.user_id, messages.user_id_from, messages_user_id_to all reference user.id, hence they can’t be an unsigned INT, but has to be a signed INT.
5. Multiple occurences of user_id references user.id, so they must be INT, not INT(11). This affects the following tables: user_profiles, session (although the field is named userid in the session table, without the underscore).
6. schemas.extension_id has been changed from INT(11) to INT, since it references extensions.extension_id
7. update_sites_extensions.update_site_id changed from INT to INT(11), since it references update_sites.update_site_id, which is an INT(11). Both columns in update_sites_extensions got a NOT NULL attribute added, since they make no sense without it.

Changed column ordering

Column ordering in some tables has been changed so that foreign keys are listed just below primary keys, as I feel that this is cleaner and makes the relations easier to read. The changes were so many that I gave up trying to write a complete list.

All in all, this schema badly needs an overhaul, to say it mildly. I have written a short blog post explaining why I think the Joomla database schema smells.

Read Jays blog entry

If you have phpMyAdmin you can use the SQL-tab to run this query. If not, you can run it from the commandline as well. Here is the query I ran: (remember of course to replace table_name, column_name, find_this and replace_with_this with real data)

UPDATE table_name SET column_name = replace(column_name,"find_this","replace_with_this");

[tags]MySQL,databases,sql,query[/tags]