The Juju QA team uses juju to build the CI services that test juju. We have built 3 CIs in fact, because juju makes it easy to move services to the cloud of our choice. Our choice is driven by which cloud provides the best combination of resources, but CI has grown beyond what one cloud can provide. We want CI in all clouds to test with the resources they provide. While juju doesn't support cross environment relations, that hasn't stopped the QA team from building an hybrid environment that straddles two clouds and a private network.

The steps to provision a machine on another network are codified in our add-remote-machine.bash script. The Juju QA team uses this script to add physical machines, machines in kvm, and machines launched in other clouds into our Juju environment. Once a machine is registered with a juju environment, any charmed service can be deployed to it. Adding a machine to a open network is painless.

add-remote-machine.bash juju-ci3 my-keys/juju_ci_rsa 10.10.10.3
    |                          |          |              |
  script                     env-name   ssh-key   private-ip

Adding machines on restricted networks may require firewall egress changes.

Provisioning the best resources for services

Juju provides two essential devops features:

• A cloud-agnostic way to provision machines. Juju supports Azure, EC2, Joyent, MAAS, and OpenStack-based clouds like HP Cloud.
• A trivial means to deploy configured services to machines. I can put Jenkins slaves and web applications into production in minutes.

The mechanism to deploy services is so valuable, that juju provides a command to add existing machines to the environment. Existing machines often have special resources not provided by a simple cloud image. Using placement, I can deploy one or more services to each machine. For example, to add a machine that I want Jenkins to use, I would issue these commands to register machine number 2 in my environment, then deploy a jenkins slave to it, and finally configure the existing jenkins master to send work to it:

juju add-machine ssh:ubuntu@10.10.10.3
juju deploy --to 2 jenkins-slave ppc64-slave
juju add-relation jenkins-master ppc64-slave

Provisioning remote machines

The problem is that juju wants a private address when provisioning a machine. The solution to building hybrid environments composed of machines from other networks was understanding what network resources juju requires during and after provisioning. First, there are three parts to juju:

• The juju client (juju command line or juju-gui) that issues commands
• The juju state-server that manages the machines and services in the environment
• The many juju agents (1 for each machine and service) that asks the state-server for tasks

During the act of provisioning (bootstrap or add-machine) the client acts as a bridge between the state-server and the remote machine. The private address is thus private to the host that the client is running on! Once the agent is running on the remote machine, it will talk to the state-server using the state-server's private DNS name. The state-server doesn't really know where the remote machine is, and it cannot access it.

Since I have ssh access to all the machines via their private address, I can add them to the juju environment. Even when I only know a machine's public address, I can add an ssh rule that maps a random private address to the public host. As for helping the agent talk to the state-server, a single addition to the agent's /etc/hosts is required to map the state-server's public IP address to the private DNS name.

The script checks the required connectivity. It tries to verify common addresses that juju will use, such as the location of the environment's private container, the server images, and juju tools. It is not authoritative. It will ask me to verify some connections that are not explicitly defined in the environment. It doesn't know the network requirements of the services that will be deployed to the machine. This isn't an issue for machines on open networks. Several of the Juju CI machines are on restricted networks, and I asked the IS department to allow egress to the required addresses and ports. We may improve the script's verification support as we extend the environment to HP Cloud and Joyent.

As for how Juju makes CI easier, that is the subject or a future post.