Category Archives: Launchpad

Concluding the bridging the gap theme

Many people have noticed application pages are changing on Launchpad. The pages might state that Launchpad does not know where a service is hosted, or the layout looks more like the common Launchpad design. These changes are a part of the final feature the Launchpad Registry team is contributing to the bridging the gap theme: To clearly state what Launchpad knows about were a project hosts a service.

This simple statement is a very large change to how Launchpad collects and presents information about the applications used by a project. The underlying goal is one that anyone who has ever tried to contribute to a project will sympathise with–“I want to contribute and I need to find where the project is hosted”. Launchpad aggregates project information and it hosts projects. Many projects use one or two Launchpad services like bug tracking and code hosting, but manage support or specifications elsewhere. Each service needs to tell users where or how they can accomplish their task. This also means that when Launchpad does not know, it should say so and not imply the project uses Launchpad.

This feature is also an opportunity to reconcile the differences between how applications present the same information. Once this opportunity was agreed to be in scope, I proposed the common experience for the five Launchpad applications (Answers, Blueprints, Bugs, Code, and Translations). The front pages will state whether the service is unknown (tell us), hosted externally (and tell you where), uses Launchpad (use it now), or not applicable. The pages will also explain if information may also be tracked in Ubuntu. The pages will not be indexed by search engines when the service is unknown. The layouts of the pages will be the same so that your experience using one application is applicable to all applications.

At first glance, this appears to be a feature that a team could deliver in a month, but that is not true. This is a very large change. We changed how launchpad collects information and the change had to be compatible with the current information so that the UI changes could be made without preventing you from using Launchpad. The description of 4 states for 5 apps is a gross simplification. When I say project, I mean a valid target for the applications, which could be a project, project series, project group, distribution, distro series, distro package, user, or team. This is 4 x 5 x 8 pages/conditions. The number of interaction we are building is much less than 160 pages because not all states apply to every app, nor does every kind of thing I listed. The Registry team is designing, building, and verifying 80 distinct pages. This is a lot of work for a 4 person team (this too is an exaggeration as most of my time is spent fixing oopses and planning new features). We appreciate your patience and we welcome your bug reports.

Remixing Ubuntu using Launchpad

Creating an Ubuntu remix is a big effort that requires the orchestration of a lot of small pieces. Many users assume they want to create a project or distribution to manage this, but neither can manage all the pieces. I recommend using a Project Group to manage the unique parts of the remix, and a team with a Personal Package Archive (PPA) to define the unique and common parts that are published.

Let me state first that while this scenario does describe what many call a derivative distribution, Launchpad does not provide all the features you want in a distribution. Ubuntu is the only distribution that has packages built by Launchpad. While you could have a distro whose series parent is an Ubuntu series, your distro will have no source packages, binary packages, or mirrors. The archive management tools are exclusive to Ubuntu. Translations are also exclusive to Ubuntu. This is not out of selfishness, it is just a matter of fact that these seminal services were built for Ubuntu, and extending them to support other cases is a lot of work.

You can use other Launchpad features to accomplish a remix. This practice was discovered through trial and error by users. It also reveals that in most cases, users really do not want a Launchpad distribution, because they have original work that that stands on its own.

Launchpad Project Groups are a way for a community (a team) to control several projects. The Project Groups drivers are automatically project drivers, allowing a team to plan releases across several project are once. Project Groups allow you to see a unified view the series and milestones on the sub projects.

Your projects represent the original work that makes your remix special. The original work for a project is the code, artwork, and documentation kept in the series branches. The team that owns the Project Group should also be the owner of all the Projects [1].

Most remixes need one or more projects that provide new art for boot, login, and desktop themes. Some remixes want to provide packages for projects that are not in Ubuntu yet, so they maintain a mirror of the code in a Launchpad project [2]. Some teams are using Launchpad to host their projects–it is pretty common for a team to build specialised tools then decide to distribute them using a live Ubuntu ISO as a base.

Many teams choose to create a project to represent the meta-package that controls the default mix of packages. This is an ugly hack, and the explanation deserved more than a footnote. Since the meta-package project does not produce something that a user or system runs, it does not deserve to be a project. Launchpad projects permit communities to share code, translations, etc–these projects do not have much to share. The branch that holds the code (/debian directory) of the meta package could easily be in a personal branch [3], but that prevents the team from sharing it between themselves. A project is the better choice in this situation.

Here is an illustration of the common way to set up a Project Group that distributes its packages

ProjectGroup -- owner Team -- PPA
|
+ Project1 (original artwork)
|
+ Project2 (no Ubuntu package)
|
+ Project3 (team is the upstream maintainer)
|
+ Project4 (meta-package hack)*
.
.
.

The team needs an archive to distribution its original packages and updated Ubuntu packages. Any team can create a PPA from their profile page on Launchpad. Team can dput their packages to their archive and Launchpad will build them. Users can add the PPA to their sources list, install the meta-package, and get all the packages in the remix in a single install.

Many teams want to create custom versions of Ubuntu packages. The simplest way to change a package is to get the source, add a patch, then dput it to Launchpad, but you then need way to manage that patch. An alternate way to so this is to use a source package branch. Every Ubuntu package has a branch that contains the source code, debian packaging rules, and patches to build the packages. You can usr bazaar to pull a copy of a branch, make your changes, then push them to your team’s location. Launchpad beta users are testing a new feature called recipes. They allow you to describe how to merge and nest several branches to create your source package and dput it to Launchpad.

This kind of problem also explains why users and teams own archives, not projects. The archive contains packages derived from the Project Group, from other Projects that are not packaged in Ubuntu yet, and modified packages from Ubuntu. The team’s archive is a single source that provides a diverse set or packages. The team is not the owner of the projects it packages, and it does not need to be to distribute them. Other users and teams can reuse your branches to make their own remixes. Your change may make their way into Ubuntu.

[1] Any user can make a project be a part of the Project Group you control. This is a bug. Control and affiliation are conflated in Launchpad and this needs to be fixed.

[2] You may not want to make mirrored projects a part of your group because you would be happier if someone else managed it. You can always make someone else the maintainer if a volunteer appears.

[3] Since projects are shared. there is no such thing as a personal project. You do not need a project store your personal branches. All Launchpad users have a location to keep non-project branches, but I have never seen a Launchpad page tell me about them :( Also “+junk” implies the branch has no value, which is wrong…it is versioned and public, of course it has value, it is just not a source for a project :( See https://help.launchpad.net/NonProjectBranches

I am still disappointed by Launchpad’s page headers

I recently investigated some spurious errors running some locale tests. The tests failed because there was no page title, which may come from an obsolete pagetitles module, or from the view class. Reading the source code side tracked me for an hour and I cannot get it out of my head, so I think need to write about the immediate and root issues.

There is a mechanism to fall back to a form’s label when the page title is not provided, but not every page is form. I can see in the rules that many pages really do require a property to provide the page’s heading (The h1 element) and something to provide the first item in the page title. Bread crumbs are the reversed page title, so page_title is also the last bread crumb. This is bad because many views define one property to be identical to the other, eg page_title = label. I reported a bug last year about cases where we need to do this because I was informed it was not necessary. The code clearly states page_title is required. Only 3 lines are needed to avoid defining identical properties in view classes, fixing this issue would not fully satisfy me.

The Launchpad 3.0 header is still a root problem for page headings, bread crumbs, and page titles. When the heading and the last bread crumb are the same, redundant information makes the top of the page hard to read. It looks wrong. It really is wrong, but not according the Launchpad’s header rules. In general, Launchpad should not generate redundant headings, or encourage engineers to create redundant headings.

We sometimes want the view’s h1 heading to be different from the last bread crumb because it is not clear what is being edited. The problem is most clear when you are looking at deep pages where the context object is not the branded project/distro. For example:
https://launchpad.net/ubuntu/maverick/+source/wsjt/+changelog

The heading cannot be “Change log” because seen under “Ubuntu”, the user will pause as ask “what the …?”. We use a label to clarify the Change log is for a secondary context “””Change logs for “wsjt” source package in Maverick”””.

It may be possible to drop the either the page_title or label properties if the header was redesigned to show the immediate context before the action/view that the user is seeing. For example:

Ubuntu {project}
“wsjt” source package {context}
Change log {view}
Ubuntu : Maverick (10.10) : “wsjt” source package : Change log

This suggestion is fatally flawed of course because it does not consider the Launchpad applications (bugs, code, …)

Essential Source Package information

Have you ever tried to register a project for a source package in launchpad? You probably wanted to locate the upstream bug tracker or get the latest code. You were probably thinking that Launchpad has a lot of the information. You could copy some information from the package, then add the repository and bug tracking information. Isn’t it hard?

My team have discussed adding a drive-by-project-registration for months, but we still cannot provide the information needed to register a project. The project registration form asks for a description and licenses, but this information is not presented in Launchpad. I know every Ubuntu source package has a description and it cannot be in one the Ubuntu’s repos without a copyright file.

This package is listed on the first page of packages that have bugs that need forwarding to an upstream project: /ubuntu/maverick/+source/wsjt

What is it? What are its licenses? I do not know how to answer these questions using launchpad. When I need to register an upstream project in Launchpad, I use packages.ubuntu.com. It allows me to search for a source package, shows me the description and has a link to the copyright file. The copyright file often has the project home page listed too.

I created a proposal to change Launchpad to provide this information so that it is easy to answer questions about packages and register projects for them: Essential Source Package Information

Launchpad bug tracker paper cuts

I have been doing a lot of work in the Launchpad bug tracker recently. Reading bug reports and code remind me of my own frustrations using the Launchpad bug tracker.

Bug reports include deactivated projects. I have seen bugs that affect more than one project, and one is deactivated. I get a 404 when I navigate the link. No website should be making dead links to itself. Surely the fix is just a few lines to filter out deactivated projects.

I cannot change the affected project to a distro and source package in a bug report. I can add a distro, but that does not remove the bug from the project. The project’s bug team will get irrelevant email forever. I get a lot mail that is not about my projects. One frustrated user created the “null” project that anyone can reassign these bugs to. This is wrong from Launchpad’s perspective, but even I use the “null” project because the level of email is a serious hindrance to using Launchpad. The Launchpad Answer tracker lets me change the question from a project to a distro, why can’t Launchpad Bug tracker do the same?

Owners of projects are sent bug mail when the project does not use Launchpad bugs. WTF? Launchpad mailing rules fall back to the project owner if the bug supervisor is not set. The rules never check if the someone explicitly set that Launchpad is used for bug tracking. That is in sane. This is a contributing factor to why users complain about too much mail. This is also the reason you cannot contact the Registry Administrators team; their email address is never checked because it is filled with bugs reported about Fedora and Gentoo…which do not use Launchpad.

I do not understand bug nominations. Every one I have seen appears to be made by a user who thinks he can vote for a low priority bug to be fixed in a development series. I have even seen trunk series nominated, though it will never be released. As a release manager, I use priority as my guide to planning fixes. Open source development is not a democratic process. The only use case I can imagine for bug nominations is in super large projects like Ubuntu that have too many high priority bugs. The project has a small team of release managers and a large team of project drivers that need to select candidates for the release. If Launchpad must have nomination, the feature should be restricted to project drivers.

The road to failure is paved with bugs targeted to series. I advise everyone to never target a bug to be fixed in a development series because you cannot undo your mistakes. The action creates a conjoined task to the series. You cannot delete it, you cannot change the series it affects, and it blocks you from updating the bug in the project or distro. Since the bug tasks cannot be removed, Ubuntu has hundreds of bugs targeted to obsolete series that will never be fixed; its bug listings are lies. I recommend creating a milestone without an expected date for the series. Target bugs to that milestone, you can change your plans and update your bugs as needed.

Speeding up page loads

I have been fixing Launchpad page timeouts. This is traditionally solved by optimising queries, but in the case of timeouts related to milestones, the problem in in the Python code. The primary reason milestones are slow is that bugs are shown or summarised, which requires repetitive permission checks. The secondary reason is the number of bugs and blueprints that are shown so that the release manager can make informed decisions.

I choose two approaches to solve the two problems. In the case of bugs and bug tasks, I applied a faster permission checker on they after first verifying that the user had view permissions for all of them. For listing large chunks of content, I used the new memcached tales directives written by Stuart. They are easy to use, but some fore-though is needed because you must understand how the content will vary when viewed by different users.

In my first pass, I used <tal:cache content="cache:public, 1 hour"/> to wrap content that did not contain private data so that all users saw the same chunk. I used <tal:cache content="cache:private, 1 hour"/> to wrap bugs. This gives each user a a cached view and all anonymous users a shared cached view. After a few test failures, I realised I also needed to use the private rule for chunks that may contain links to modify objects; milestone listing often contain links that encourage the release manager to provide information.

Browser tests may also require you to uses MemcacheLayer.purge() to verify the changes users will see when cache expires. While you can clear the memcached when content changes, there is API (yet) to clear all page caches that involve the modified object.

With a few deft lines of code, I was able to speed up milestone and series pages. I had also made projects and distro pages faster since they share portlets with series. I decided to extend the scope of my branch to cache the public content in all the portlets used by distros, projects, and project groups. You may need to wait an hour to see the latest bug reported in the Ubuntu main page, or see a bug appear on the milestone page, but the pages will load faster, which is better for most users.

Being a Launchpad Registry Admin

Over the past few months, it occurred to me that by a confluence of Launchpad responsibilities, I am the God-Emperor of Launchpad Registry Admins. I never intended to take this role. I certainly did not covet it. I just realised that there were a lot of data issues in Launchpad that I personally had the power to fix.

One, Launchpad engineers are members of the Registry Admin team because we are often requested to update projects that do have a Launchpad maintainer (those projects owned by Registry Admins). We are thus, the itinerant maintainers of more than 1500 projects. Projects that were missing license information, missing source code, and missing upstream bug tracking information. These are projects that need someone to adopt, keep the data current, and ensure it can be used by any Launchpad community.

Two, Most Launchpad questions are about project and user management, which is the domain of the Launchpad registry app, and I am the only answer contact. When you want a registry admin project updated, or you want to adopt it, I am the person who helps you. I work with all project and team issues. So while all Launchpad engineers are Registry Admins, I am the person who uses the responsibility most often.

Three, I review every registered project, checking that it is legitimate and has proper licenses. Many new projects are gifted to the Registry Admin team, and I provide the missing information. I review 25 or more registered projects every day. I reviewed the 4000+ backlog of projects last year.

Four, The project review form is insane, and it would be easier for me to see the state of all projects if all projects have licenses…so I set the license information the Registry Admin owned projects. And since I was looking at the project page, I updated the home page link and linked the project to an Ubuntu package when I could. I fixed about 1000 projects.

Proposed Launchpad bug configuration page

The registry team is going to work on some long standing issues in the Launchpad bug domain. We are going to create a single page to configure the handling for projects. We have a draft of the proposed changes that shows that the bug expiration and remote project fields are subordinate to the choice of bug tracker. We need to revise labels and descriptions of the fields to make their purpose clear.

Proposed Launchpad bug configuration page

Proposed Launchpad bug configuration page

We intend to make it easier to locate a registered bug tracker by using to a lazr-js picker instead of a selection list with 700 trackers. You will also be able to register a new bug tracker at that moment. You will never have to find where the bug tracker list is and how to add a new one again.

Proposed Launchpad bug tracker registration overlay

Proposed Launchpad bug tracker registration overlay

Launchpad CSS is like a Jackson Pollok painting

I was migrating old CSS styles to Launchpads YUI-based style sheet a few days ago. I became lost in the new style sheet, and I was confused by what appeared to be duplicate or contradictory styles. I spent my evening reordering the styles by selector types instead of by application uses
When I was done I discovered we have several table styles that do not look like the intended style of the site. There are more than 10 list styles that imply the transitions to sprites is incomplete.

I think the decision to have sections for applications was wrong. It encouraged developers to create styles that are not reusable. Developers should be updating the styles so that all Launchpad evolves. (I recommended the continuance of this rule when I created the YUI style sheet.) The YUI style sheet is ridiculously large now, and it is only 9 months old. This is a strong indication that there are many exceptions in the presentation that users must learn.

My next round of changes will be to normalise the sheet to remove the duplication. I added a CSS reformatter and validator to gedit-developer-plugins to help me do this. After that, I think I need to reduce the number of classes by half, which also means updating some old layouts. There is still an unresolved issue of application specific styles in the old and new style sheet. I think a lot of work is needed to remove the exceptions from the pages. Doing this has one great advantage–future redesigns will require CSS and base template changes instead of changes to every template.

Hacking on trivial Launchpad bugs

The Launchpad registry team has a zero-trivial bugs policy. Last year we observed that trivial bugs, though they are often low priority, they have the highest certainty to be fixed and are very visible to users. We decided that since an engineer can close 8 bugs in a day of work, there should never be more than 8 trivial bugs in the project.

I had a lot of meeting this week. I was too distracted to commit to working on a bug issue. I decided to start a branch that I could fix trivial bugs in between meetings. After a day of meeting, I had a branch that was ready for review.

We I say trivial bugs have the highest certainty to be fixed, I am referring to my own primer on bug triage. There is almost no risk in fixing the bug, and since we will know in an hour instead of days whether our efforts fixed the issue, trivial bugs have an intrinsic high priority. I really cannot say every trivial bug is high priority, but when I consider several of them together, they are high priority.

Most trivial bugs are grammatical errors, presentation issues, or navigation issues that do not stop users from accomplishing their task. Trivial bugs are obviously easy to fix, and users are reminded of them every time they repeat their task. These bugs are very annoying, they amplify the user’s perception of poor quality. Engineers can make a big improvement in user experience by fixing trivial bugs.