Devops and RPM Distribution

Regularly deploying and provisioning RPMs is increasingly becoming a common need. With devops automating every aspect of deployment, including setting up new hosts in virtualized environments, packaging systems such as RPM are getting more and more important. Managed RPMs provide a great level of control over installations and easy upgradability.
Amazon’s own AWS images are one good example where RPMs deployed to a controlled YUM repository can provide a great experience for users. Yet, automating the creation of a good YUM repository for sharing and controlling RPMs is a bit tricky.

A Look at Java Libraries Provisioning

Provisioning and resolving Java artifacts is a long-time common practice. All build tools (Maven, Gradle, SBT, Ivy, etc.) can resolve dependencies from binary artifact repositories and deploy back the artifacts they have built. These artifacts are later consumed by other builds and packaging tools. With CI servers, this process is repeated concurrently and more often, automating build pipelines where artifacts are exchanged between different build stages through the repository.
So, while we have achieved a standard nice automation for creating and provisioning module artifacts (such as, Maven artifacts), this is not the case for system package modules, such as the ones created by RPM or Dpkg.

Is RPM Provisioning Really Any Different?

One can argue that system packages differ a lot form language-level modules. One big difference is the former are focused on providing a *single*, coherent view for all installed modules. This is why conflicting module versions is problematic, why upgrade/uninstall is a must have and also why these systems employ complex cross-module semantics. Things are much easier for Java applications in this sense, where every application sees an isolated view of modules.
However, in terms of sharing and provisioning artifacts through a shared repository, the basic principals are essentially the same!
  • Module artifacts are packaged and deployed into a remote repository.
  • Deployment generates repository-level metadata/indexes that make it simpler and safer for clients to discover modules and consume them. Some examples of such metadata are: checksum files, maven-metadata providing information about the artifact versions, repository data describing the RPMs in a YUM repository, etc.
  • Clients contact the repository, often via HTTP, to download artifacts.

Making RPM Provisioning Easy

Deploying Java artifacts to a repository is a breeze and is natively supported by build tools and CI servers. Unfortunately, the same cannot be said about RPM deployment -
After building each RPM, the client needs to create the YUM repository metadata for the whole repository. This is done manually on the server file-system, and is not a by-product of simply deploying the RPM artifact. But there is really no reason for separating deployment from updating the YUM repository!
To automate the process, one would have to trigger execution of the “createrepo” command on the RPM repository after each deployment. It can be done, but requires setting up some home-grown solution.
Another approach is to introduce this functionality directly into the artifacts repository!
This is exactly the approach taken by JFrog’s Artifactory Pro. Upon deployment, Artifactory generates the YUM metadata on the fly and make it available for provisioning almost instantly.
There are a couple of huge advantages to this approach:
  1. RPMs get all the advantages of being hosted in a binary repository: they become searchable and you can apply security and visibility rules to RPMs
  2. No setting-up a separate process is needed - you are already running a repository manager, so why not let the repository take care of YUM RPM hosting automatically?
  3. The whole process is pure-Java and can be run on any OS - not just ones that support the ‘createrepo’ binary
  4. Detailed RPM information can be viewed from Artiafctory’s UI
  5. RPM calculation can be triggered automatically in the background, or via REST
  6. You can auto-generate metadata in sub-paths. This is a common requirement for repositories grouping artifacts by version and architecture into individual YUM sub-repositories.
This makes the experience of provisioning RPMs via a YUM repository as easy as sharing Java artifacts in a cross-platform way, fully automated from build tools!
To learn more about the integrated YUM repository support in Artifactory, please visit this link.
To learn even more about RPM provisioning as part of devops automation, and about other practical aspects of devops in the cloud, register yourself to the “Devops in the Cloud” event, hosted by JFrog at Neflix, on Thursday, December 15, 2011, presenting the cloud experience of Netflix, CloudBees and JFrog. See you there!