Tag Archives: Launchpad

Common mistakes that broke Launchpad’s browser compatibility

At this moment, Launchpad is providing all features to Opera, Internet Explorer (IE), and many other browsers. Many features like bug listings, bug tag auto-completion, changing bug status and privacy, choosing people, and editing subscriptions were broken in many browsers. Some features were broken for years because developers mistook their inexperience for something broken these browsers.

I am summarising what was wrong with Launchpad and how to avoid repeating the mistakes of the past. Launchpad reviewers, I urge you to read this so that bad code is not merged into Launchpad’s trunk again. Launchpad engineers commonly work with Firefox and Chromium, two browsers with great support the latests technologies, These browsers are also very good at correcting mistakes in markup, CSS, and JavaScript. The lesser browsers are less forgiving; developers must fix their mistakes and on rare occasion write alternate code to make a feature work. Launchpad developers were too quick to blame the browser. They failed to recognise their own inexperience, and they failed to find the root cause of the problem.

Why Do We Want to Support More Browsers?

One user, lazarus, responded to my A tale of two travesties post asking “why bother with IE? Those 4% of users should switch to a *DECENT* browser.”

Canonical’s partners often run older releases of Firefox and Chromium. Some partners are running IE. Their staff cannot install software onto their computers, they have no choice but to use the corporate browser. The people who help build Ubuntu, who help Canonical enable Ubuntu to run on new hardware, cannot use Launchpad to do their job.

Consider users who use their phone or tablet with Launchpad. These browsers are also locked down, and they are commonly older versions of browsers. There were several bugs in the past that prevented Canonical staff from using their phones to login to Launchpad. They, even myself, had to find another computer to respond to a notification received via phone. Mobiles devices are now commonly used to work on the Net; Launchpad risks loosing users and projects who can choose another service.

Servers often do not have a GUI. We assume Ubuntu server is running headless. Servers also tend to be LTS installs that might can be 2 to 5 years old. Tasks such as reporting a bugs, updating it, and adding a comment must work in a text-based browser without JavaScript.

What is the Cost of Supporting More Browsers?

Again lazarus stated in his reply to my struggle to run IE in Ubuntu, “I’m really sick of seeing all those work-arounds just to make it work for that damn browser.”

I wrote less than 20 lines of code to Support IE, yet I removed more than than 125 lines that disabled it or other browsers. I Also removed about 200 lines of duplicate or alternate code from <noscript> elements. It takes fewer lines of code to support older and mediocre browsers than to not support them!

For every bug that was tagged as affecting IE, I discovered it also affected Opera, Konqueror, Firefox 3, and even the latest version of chromium. Fixing one bug solved an issue for multiple browsers each time. I found many bugs overlapped with or were duplicates of other bugs that were listed as a higher priority, yet remained unsolved. Understanding the root cause of a problem allows us to make a single change that fixes many things. Honestly, the most common way to fix an issue was to read the YUI documentation to learn what Launchpad was doing wrong. Thus, the root cause of IE not working is rarely IE itself…the JavaScript bugs in Launchpad are caused by inexperience.

Running other browsers can be difficult as I reported in my Travesty post. I now have a custom apache configuration that cannot be copied to other developer machines. Running Microsoft’s official IE developer images take all my available RAM. I also run the VM image from an SD card because Windows is still bloatware. Installing Konqueror and all the KDE dependencies can also be a problem if you are short on disk space.

The maintenance savings by writing good code that supports all browsers are diminished by the days or weeks it takes to install all the browsers to reproduce a bug. If I cannot reproduce a bug, I cannot fix it. For many Launchpad developers, the best way to not fix a bug in a mediocre browser is to avoid introducing the bug. This is why we use YUI as our JS library, it reconciles the differences between browsers. This is why we have code standards and reviews, to ensure that we agree the code is correct and maintainable. We want to recognise bad code before it is merged into trunk and we absolutely must avoid dangerous code,

The Last Sign of Inexperience

This example of code is dangerous and unmaintainable:

if (Y.UA.ie) {

This is the last sign of inexperience. Bugs have already been slipped past review, merged into trunk, and are probably in production preventing user from completing their task. Why is IE not using the feature? Which browser versions and YUI versions were tested? This kind of guard often has an associated comment like:

# Usual IE brokenness.

I think the comment can also be interpreted to mean: “I do not have IE installed, I do not trust it, I do not trust my own code, I do not understand YUI”.

Do not let this kind of guard merge. If one browser does not work, there are probably 5 other browsers, like those shipped with an Ubuntu LTS, that also do not work. This branch needs fixing!

Is This a Presentation Problem?

Several bugs fixed by William Grant and myself were not JavaScript bugs as was reported. They were CSS or markup bugs. These bugs were trivially fixed in minutes when shown to the right person, yet some of these bugs were reported years ago. Your chances of fixing a presentation problem with JavaScript is similar to painting a house with a hammer. If the UI for a JavaScript feature does not look right, or not render at all, consider a CSS or markup change.

Non-conformant CSS the most common cause of broken presentation. I often uses Wikipedia’s Comparison of Layout Engines when researching differences in layout. Even the latest versions of Firefox and Chromium have inherited defects from their layout engine. Firefox was showing a scrollbar because the form overlay buttons had a text-indent property set to twice the width of the page. Though Chromium did not show the scrollbar, I think Firefox was right in this case; I fixed the issue by setting a text indent that was a few characters wide. Many issues can be fixed using alternate properties or additional CSS selectors to add properties to the problem elements. For example, IE has poor support for visibility (introduced by Netscape in 1998), but has better support for display (introduced by Microsoft in 1998). IE was always showing the fields in the overlay forms that change the page. Though the form overlays were visibility: hidden, IE requires a specific selector for .yui3-lazr-formoverlay-hidden input to be display: none.

Experimental CSS can be both the cause and the solution to a display problem. We want Launchpad to look good, and we often use CSS3 to define the styles. Many CSS3 properties are not fully implemented in browsers, but there are vendor properties (ones with a dash prefix) that do support the presentation. Launchpad has many of these for Gecko and Webkit. We also need to add them for KHtml, Opera and IE. While IE does not support the CSS3 box-shadow property, it has supported shadows since 1998 and we added its vendor property to give all overlays shadows.

Invalid Markup breaks older browsers, but newer browsers are better at fixing (and hiding) markup problems. You can read about how browsers work in depth, but I think I can summarise the issue in a few sentences. When browsers discover an error while parsing the markup, they pause to fix the markup then carry on. For example, paragraphs cannot contain lists. Most browsers assume that the author meant that there is a paragraph before and after the list. If you query the DOM using JS, you may discover it does not match the source. When I switched the Launchpad test suite to use a webkit-based test browser many tests failed. Developers were ignoring test failures because they passed in Firefox. I traced the errors to existing bug reports and discovered code was creating invalid tables and Chromium fixed the markup differently than Firefox. IE is very prone to choke on malformed tables. The lesson is clear, write what you mean, or else browsers are free to reinterpret your bogus markup and topple your scripts.

Launchpad users will soon loose their last line of defence against invalid markup when Launchpad switches to Ubuntu 12.04. My squad stopped development a few months ago to learn why Launchpad’s ec2 test runner failed a test that passed on all the other test runners. The reason is because the script used bad markup that toppled the Ubuntu 10.04 version of my test browser. The new webkit library in Ubuntu 12.04 is better at correcting markup, hiding from mistakes from tests, allowing the mistakes to be released to break older browsers.

Is This a Structural Problem?

Structural problems are issues where the choice in markup is brittle or unmaintainable.

Noscript is BAD. Netscape meant well by introducing the <noscript> element, but it leads to maintenance burdens and it assumes that all scripts run perfectly. The intent is to have two implementations and do not diverge. They do diverge though, leading to differing and confusing behaviours. If the script fails to initialise, the user gets no behaviour. This situation was exacerbated in Launchpad with the use of the if (Y.UA.ie) guards that meant IE users never saw the HTML or JavaScript links to change project roles or change blueprints. Launchpad prefers, and YUI recommends, progressive enhancement; write simple markup that does not need JavaScript, then modify the page to add the scripted features. Scripts that replace or hide the HTML feature should do that as the last step; If the script fails, there is still a usable link.

Mutating parent markup while parsing is dangerous. Several widgets like the TextAreaEditor were mutating their parent element before the parent element had closed itself. Only the mad and the brave mutate the thing they are iterating over. Chromium and Firefox handled the insane situation with grace, but other browsers threw a wobbly. IE 8 could not render the bug page, it stopped at the bug description. This situation is hard to see in a review. In general the script block must be outside of the elements it changes, but several widgets enhance parent and grandparent nodes, which means the markup appears to have redundantly nested elements. Some of this madness can be averted by running the script on DOMReady. There are many scripts that load as the page is parsed; I suspect these could all be changed to run at DOMReady.

Is This a YUI Problem?

YUI also has bugs. Has this issue been reported in YUI’s bug tracker? Which version of YUI is Launchpad using? Does upgrading fix the issue? Maybe an issue needs to be reported?

There were several hacks in the Launchpad bug pages to load scripts during parsing or after the load event because DOMReady was buggy during YUI3 preview releases. Both Opera and IE had difficulty during running scripts during parse time, and running the scripts after the load event caused the page to flash. Launchpad has been running YUI3+ for almost two years, the hacks should have been removed. The code required a proper XXX comment to remind developers to update the scripts.

# XXX: 2010-03-12 bingo bug=234567:
# DOMReady is bug in YUI3 PR3. Test DOMReady again in YUI 3.0.

There was also a case where Launchpad ran a version of YUI that miswired the event handling in IE. Several if (Y.UA.ie) guards were added without a comment explaining the issue, nor a stating the bug number in YUI’s bug tracker. I do not know when when these issues were fixed, but they are fixed in YUI 3.3.

Is This a EcmaScript or DOM Problem?

Launchpad was created in the age of EcmaScript 3 (ES3). This is now the age of ES5, and Launchpad developers are using ES5 browsers like Firefox and Chromium. Browser that the developers once used to build launchpad stopped working. Launchpad stopped working with the browsers available to Ubuntu LTS such as 08.04. Launchpad developers use jslint and YUI to ensure that code works in old and new browsers.

Was the lint report ignored? Launchpad’s make lint command uses jslint to prevent known issues before they are merged into trunk. There are still a lot is issues reported in the old lazr modules that need fixing. We fixed coercion mistakes and unnested functions created in loops to address bugs in older browsers.

Which version of EcmaScript? There were a lot of failures caused by unsafe ES5 calls. William fixed many script failures in older browsers by ensuring that Y.Array() is called properly. It is very important to understand that Y.Array() returns a native array. This call is pointless:

var finklesnarks = Y.Array([]);

This call is dangerous because it creates a native ES3 array in older browsers, then calls an ES5 method:


I was the author of some of these bad calls. The safe way to use ES5 array with YUI is:

Y.Array.indexOf(finklesnarks, fnord);

Are events happening in a different order? Are links doing both script actions and standard actions? YUI docs describe event bubbling, capturing is never mentioned. I thought this meant that YUI had disabled the capture phase of event propagation, simplifying how we work with events. No, this is not the case. Firefox and Chromium still prefer capture propagation, and IE uses bubbling propagation exclusively. Actions that work in the former browsers often have secondary, and wrong, actions happening in IE. This is because Firefox and Chromium are done propagating, but IE is just starting. Most Launchpad click handlers use e.preventDefault(), which means “stop the default action, continue propagating”. IE does continue propagating and unexpected things happen. A lot of Launchpad links work more by accident rather than by design. The correct method to call is e.halt() which stops the default action and propagation. We want to use halt() most of the time. In cases where we want multiple subscribers to respond to an event, we are using a custom event.

Does YUI docs demonstrate that the code will work? This is an odd point to check, but a lot of code was merged into Launchpad trunk without regard to YUI documentation. The bug status widgets did not appear on click in some browsers because they are designed to only display if you used the left mouse button. But YUI documentation states that the button can only be known in the mouseup event…the mouse property explicitly states that it is not for use with click. Removing the unsupported check of event.button fixed a bug that has been open for 3 years.

Is This a Broken Browser Problem?

These are the cases where developers must write code to reconcile the differences between browser implementations, or choose to not support the browser.

Non-conformant surprises are cases where browsers claim do support a feature but do not. Launchpad new bug listing UI depends on history management to sort columns and show batches. History management depends on URLs, but IE’s Link.pathname is missing the leading slash so URLs were not relative from root. The fix is trivial once you see the invalid paths in Launchpad’s oops reports and then look at the code that makes URLs. As we do in Launchpad’s python code, the method that wraps the pathname ensures there is a leading slash.

Alternate API nuisances are cases where the browser doesn’t support the methods the script uses, and neither does YUI. The bug tag autocomplete feature never worked in IE, but there was code checking for it. I first refactored the method to check for the methods Launchpad prefers, then checks for the alternate methods. This was hard to fix, I spent an hour reading about how IE creates selection ranges before I could write a 5-line block to do what element.endSelection does.

Unsupported nightmares are cases where neither the browser or YUI support the method. There is one case in Launchpad were this happens. The series timelines use canvas to draw the series, milestones, and releases. It was disabled for all IE version, but IE 9 does support canvas if the page states it is HTML5 . Instead of checking for browser version, I created a method that checks if the browser can create the canvas, and if so, the series timeline is rendered. IE 8 and below do not get the interactive image. Other browsers that I have not planned for will show the canvas if they pass the check.

How do I ensure a browser falls back to working code?

At this moment, there is only one guard that prevents KHtml browsers using a feature. There might come a time to add more guards because Launchpad, YUI, and a kind of browser do not work, and no one has time to make it work. We need to know the browser version that was tested and the version of YUI, for example:

if (Y.UI.ie < 6 && Y.version <= 3.3)

I think we want several experienced YUI developers to agree that this is the proper course before giving up though. When a new version of the browser or YUI is released, we know to check the feature. Does the guard need to be revised, or can it be removed?

Launchpad font-size broken by design

I looked at Launchpad bug #523189 today. This bug cannot be fixed until Launchpad abandoned YUI-Font rules — percentages have unpredictable results when pages are constructed from fragment. Canonical own Web guidelines specify the use of pixels.

Launchpad uses YUI-Font to normaize font sizes across browser and OSes. It stipulates that all font sizes must be expressed in specific precentages to get the expected point size. This is difficult to use because browsers know that percentages are relative; nesting classes have surprising results.

The following text uses nesting with the same percentages that Launchpad uses.

An example of broken font-sizes in Launchpad

An example of broken font-sizes in Launchpad

Launchpd pages are constructed from many html fragments. The fragments are intended to be usable in many places on a page and display the the same way. Launchpad cannot use percentages for font-size.

Speeding up page loads

I have been fixing Launchpad page timeouts. This is traditionally solved by optimising queries, but in the case of timeouts related to milestones, the problem in in the Python code. The primary reason milestones are slow is that bugs are shown or summarised, which requires repetitive permission checks. The secondary reason is the number of bugs and blueprints that are shown so that the release manager can make informed decisions.

I choose two approaches to solve the two problems. In the case of bugs and bug tasks, I applied a faster permission checker on they after first verifying that the user had view permissions for all of them. For listing large chunks of content, I used the new memcached tales directives written by Stuart. They are easy to use, but some fore-though is needed because you must understand how the content will vary when viewed by different users.

In my first pass, I used <tal:cache content="cache:public, 1 hour"/> to wrap content that did not contain private data so that all users saw the same chunk. I used <tal:cache content="cache:private, 1 hour"/> to wrap bugs. This gives each user a a cached view and all anonymous users a shared cached view. After a few test failures, I realised I also needed to use the private rule for chunks that may contain links to modify objects; milestone listing often contain links that encourage the release manager to provide information.

Browser tests may also require you to uses MemcacheLayer.purge() to verify the changes users will see when cache expires. While you can clear the memcached when content changes, there is API (yet) to clear all page caches that involve the modified object.

With a few deft lines of code, I was able to speed up milestone and series pages. I had also made projects and distro pages faster since they share portlets with series. I decided to extend the scope of my branch to cache the public content in all the portlets used by distros, projects, and project groups. You may need to wait an hour to see the latest bug reported in the Ubuntu main page, or see a bug appear on the milestone page, but the pages will load faster, which is better for most users.

Being a Launchpad Registry Admin

Over the past few months, it occurred to me that by a confluence of Launchpad responsibilities, I am the God-Emperor of Launchpad Registry Admins. I never intended to take this role. I certainly did not covet it. I just realised that there were a lot of data issues in Launchpad that I personally had the power to fix.

One, Launchpad engineers are members of the Registry Admin team because we are often requested to update projects that do have a Launchpad maintainer (those projects owned by Registry Admins). We are thus, the itinerant maintainers of more than 1500 projects. Projects that were missing license information, missing source code, and missing upstream bug tracking information. These are projects that need someone to adopt, keep the data current, and ensure it can be used by any Launchpad community.

Two, Most Launchpad questions are about project and user management, which is the domain of the Launchpad registry app, and I am the only answer contact. When you want a registry admin project updated, or you want to adopt it, I am the person who helps you. I work with all project and team issues. So while all Launchpad engineers are Registry Admins, I am the person who uses the responsibility most often.

Three, I review every registered project, checking that it is legitimate and has proper licenses. Many new projects are gifted to the Registry Admin team, and I provide the missing information. I review 25 or more registered projects every day. I reviewed the 4000+ backlog of projects last year.

Four, The project review form is insane, and it would be easier for me to see the state of all projects if all projects have licenses…so I set the license information the Registry Admin owned projects. And since I was looking at the project page, I updated the home page link and linked the project to an Ubuntu package when I could. I fixed about 1000 projects.

Fixing project licenses using the suggested Ubuntu packages portlet

On Sunday I did an experiment to see how quickly I could provide the license information for old projects owned by the Registry Admins. There were 768 projects that I had the power to fix. Locating the license information is hard if you only know the upstream project. The copyright file in debian source package has all the license info, but I do not always know the source package (if there is one). There is a new feature on edge that suggests the Ubuntu package, and that makes this task much easier.

Reviewing a project takes 15 seconds. Linking the project to an Ubuntu package, providing a license, and reviewing the project takes 4 minutes. Some projects that are already linked to a package can be done in 1 minute. It may take 10 minutes if Launchpad suggests many source packages. I need to view the latest version number of each suggested source package in Lucid, and search http://packages.ubuntu.com/ to verify the version and description match the project and candidate package. I disabled a few projects because there was already a registered project with license information, and already linked to a package–I assume the project I was looking at was named differently from the package, so users re-registered it.

It will take me about 3 weeks to fix the remaining projects. That will still leave 4000 projects in Launchpad that are missing license information. I need to think of a strategy to get those fixed.

The benefits of oops reports and test driven development

Today I fixed a bug where users could not link a project series to package in an older Ubuntu series. I reported the bug when working with Javier to clean up the duplicate glade projects registered in Launchpad. We both got oopses when we tried to fix the packaging links from the project’s packaging form. We fix them using the distro series packaging form.

I assumed the project version of the form was ignoring the submitted series; it always uses the current Ubuntu series. I wrote my test using the same situation that I experienced, and I expected a failure when I ran the test. I did get a failure, but it was not about the current Ubuntu development series. It was a database IntegrityError. I know that error!

Every morning, I read the oops reports from the many Launchpad process. Every exception and timeout in Launchpad is collated into a a daily report for the engineers to read. I report bugs for each problem that relates to the registry application. Most bugs I report from oopses contain enough information to fix the problem within a week. Last week I saw an IntegrityError in the oopses, but I could not reproduce it. So while I scheduled the bug to be fixed in the 10.04 release, I could not write a test it to fix it. I could not see that the IntegrityError was about a series different from the one the user submitted.

I got lucky today. The test setup creates all the objects in play. My bug from yesterday, and the bug from last week were the same bug that manifests itself in two ways depending on the state of the package in the current series. I marked the IntegrityError bug to be a duplicate of the bug I was fixing. The form was indeed ignoring the submitted distro series; it was a one line fix. If I had written the fix before the test, I would still be investigating the oops without a clue.

Fixing the bug tracker widget

The Launchpad registry had a sprint in March 2010 to solve project registration issues. The topic was broadly set to be “drive through project creation”, but the goal was to allow a community to provide project information at the time of need. For example, a user wants to report a bug about a package in Ubuntu. The bug must be forwarded to the upstream project to be fixed. The user may need to provide the upstream project’s bug tracking information, and possible register the project in Launchpad first.

I talked about the madness of registering a bug tracker yesterday. Launchpad has many registered project without bug tracker information because there is no single simple page that allows a user to provide what is needed. I did a spike and could see that while we can collect the required information together in a single page, the information that the user must provide is not clear.

broken bug tracker widget
The disorganized information needed to configure a bug tracker.

Look at the picture. Can you tell that the “Remote project” field is subordinate to “In a registered bug tracker”? Bug expiration and reporting rules are subordinate to “In launchpad”. I still cannot register a new bug tracker from this page, not can I set the bug supervisor (upstream bug contact), or security contact. The bug tracker widget needs a rethink, and I suspect a model change is needed, because the logic in the view is representing a boolean field and a bug tracker field as four radio buttons. These is some zany logic trying to reconcile the states.

I do not anticipate us starting on the new widget for a few weeks, but Edwin did take a partial step by extracting the bug tracker information from the project’s +edit page. We will update the new page to be the single place that a user must visit to configure a project’s bug tracker and contacts.

The collective madness of setting the bug tracker for an upstream project.

This IRC transcript illustrates the pain of setting the upstream bug tracker, and the agony of defeat when you succeed.

<bac> sinzui: Remind me how do you set/determine the bug tracker for a project?
https://launchpad.net/ubuntu/lucid/+source/claws-mail shows one but i cannot find it on the project.

  1. Visit the front page of Launchpad Bugs
    <bac> check
  2. Follow the link to Bug trackers at the bottom of the page
  3. Locate the (+) Register another bug tracker link at the bottom of the list of despair
    <bac> wait, there it is already in the list
  4. Register the bug tracker and remember the name
  5. Return to the project and follow the Change details link.
  6. (Be sitting down for this), Select the new bug tracker from the list of despair
  7. (optional) you may need to set the Remote project field with the ID of this project on its remote bug tracker.

<bac> Right, so if you go to https://bugs.launchpad.net/claws-mail you don’t see any mention that we know about the external tracker i knew it was bad but only anecdotally.

<sinzui> Yep. Weep at the madness. Good luck forwarding that bug upstream.

<bac> I don’t really have a bug. I just linked the claws-mail source package as an exercise and noted that the bug tracker was purported to be set.

<sinzui> My favorite part is that I have to claim that bugs are tracked in Launchpad so that I can set the bug supervisor, which is also the upstream bug contact role.

<bac> ewww

<sinzui> Launchpad can be a collective madness sometimes

No project is an island

Launchpad is often perceived to be a project hosting service. Some perceive Launchpad is a directory of packages in Ubuntu. Both views are incomplete and often lead to misunderstandings about how Launchpad can be used, and buy whom. I think the best summary of what Launchpad does is that it hosts open source communities.

From it inception, Launchpad strives to connect communities by lowering the barrier to make contributions to projects. Launchpad does not give any community exclusive control of a project or its parts. It is common to think of a project as being under the exclusive control of one community, and many services model that concept. The truth is more complicated. Successful open source projects often represent the work of many communities. Translators are interested in making all applications use their native language [1] so they need access to many projects. Users who support applications need to relate questions to answers, FAQs, and bugs. Distribution like Ubuntu need to know the bug tracker and the source code for every project who’s work is packaged.

Projects in Launchpad are more like places where communities meet to gather and share information. The communities provide bugs reports, patches, and translations to projects. Communities use Launchpad to learn where bugs are fixed, where the latest code is, what versions of the project’s work are available. Even if the project’s development community does not use Launchpad’s services, they can still use Launchpad to learn what other communities know and are doing.

[1] Accept for the Esperanto freaks ;)

Deleting team email addresses

I think the next bug for me to work on after hours is
bug #250103 [Teams cannot delete old email addresses]

This causes users and myself a lot of problems. While you can change a team email address in Launchpad, it never deletes the old one. It is kept and it is hidden so that it can never be reused. This is very annoying because users, not team own the address, so the user can never reclaim his address once he gives it to a team. This leads to a lot of requests to delete the address via SQL. This also prevents me from removing teams or their artefacts in my role as a Registry Administrator.

The right behaviour is to delete the email address when it is deactivated for the team. This is what the UI implies happens. Users can re-register the address when they choose to use it again.