• LOGIN
  • No products in the cart.

Setting up your own Package Cluster in MidnightBSD by Lucas Holt

During the development of MidnightBSD, a method was needed to test building software packages for mports developers. Packages needed to be built in a clean environment so that they would work across systems.


In 2008, The MidnightBSD project had a cluster of 21 Dell desktops available in a computer lab at Eastern Michigan University. The individual systems were slow, but there was a lot of computing power available.

Chris Reinhardt wrote the first version of Magus, the MidnightBSD package cluster, over a few months using MySQL and Perl.

The goal was to build a package on each individual computer by determining what other packages were necessary and scheduling intelligently. Each node would lock the port by writing a record to the database along with build logs and status.

Improvements to the system were developed including a web application to monitor the build, scripts to manage the build, and the ability to build in parallel on one system.

In May, Magus was ported from MySQL to PostgreSQL in order to take advantage of some performance advantages with the workload and the author’s preference with licensing.

Installing Magus

Magus requires an installation of PostgreSQL 9.2, Perl 5.14, and a checkout of mports from subversion.

To install the Magus client dependencies, install mports/ ports-mgmt/magus-utils. This includes all Perl modules needed to run Magus and the database libraries. If you wish to use the Magus web application, you should also install a web server capable of running CGI scripts such as Apache or lighttpd.

Magus source code and configuration examples are located in mports/Tools/magus.

Configuration

Before you begin configuring the system, determine how many magus nodes you will start with in your cluster, the location where packages will be written (such as an NFS share or ssh server), and which node will be the master. A master node will store the packages and mports tarball file generated by the indexing process discussed later.

Magus requires a chroot tarball containing system files. This is used to create a base directory and then during the run, a fresh build directory is created on each node. A script is provided to generate the tarball, make_chroot_ tarball.pl. You will need to generate a file for each OS version and architecture that you wish to support. The chroot file must be copied to each node inside the build directory, commonly /usr/magus.To configure PostgreSQL, create a new database called magus and appropriate user credentials. Load the sche- ma file schema.sql into PostgreSQL. Magus requires knowledge of each node that will be used to build soft-ware. Open a connection to the database with a SQL cli- ent and add rows for each node in the machines table with OS version, architecture and node name. This will allow Magus to track which machine is building which port and allow nodes to get work once started.

Screen Shot 2016-02-26 at 12.33.54

The configuration file provides the database credentials, path to package files shared by all nodes and the node name. It is used by the slave build script and all administration scripts provided with Magus. A sample configura- tion file, config.yaml.example, is provided and should be placed in the desired build location, /usr/magus/config. yaml. On each node, configure the file with the node name, database credentials and server address, path to the chroot tarball and the PkgfilesRoot path to write packages.

Starting Magus

To start the Magus cluster, a new run must be created. A run is a batch of packages and will have a unique ID. You can use the run ID with several of the administrative scripts. On the master node, run the script mports/Tools/ magus/master/update_cluster.pl to generate a new run. You will specify the OS version and architecture and a new checkout of the ports tree will be created. The generated file will be used by all the Magus nodes during the build so that they have the same version of the mports tree. You can customize the contents of the tar file as well as the chroot tar file to change how ports are built such as turning on SSL or X11 support, etc. Remember that if you add any new ports to the file, you will need to create the relevant entries in the Magus database.

Screen Shot 2016-02-26 at 12.34.04

Table 1. Magus administration scripts

To start the build process, run the script located at mports/ Tools/magus/slave/magus.pl. This step can be repeated on each node you wish to build with to complete the run.

Use the Magus web application to view current builds and progress. You will see ready ports drop to zero when the build is complete.

Generate a custom index for mport

To build a custom mport index, run the bless program with the run ID and path to the files on the master node. Bless is used to mark a Magus run as complete and ready for mass consumption. The index can be used on systems to provide a list of packages available and is required for proper operation of the mport package management tool. You can also provide a list of servers in the index file to fetch packages from over HTTP.

Future Directions

Magus has recently been ported to PostgreSQL from MySQL. We would like to create a RESTful web service for nodes to interact with the build cluster. This would allow nodes to be distributed without the need to access a database server directly.

The build software needs to be modified to disconnect fetching distribution files from the build process to avoid fetching the same source code multiple times and to speed up cluster builds.

Summary

Magus is the package building software for MidnightBSD mports. It allows system administrators managing multiple systems to generate custom packages for their own systems. It also allows developers to test changes to mports without committing code to the subversion repository.


Screen Shot 2016-02-26 at 12.34.14About the Author:

LUCAS HOLT

Lucas Holt is the founder of the MidnightBSD project and a Senior Ap- plication Programmer/Analyst for the University of Michigan in Ann Arbor, MI, USA.

0 responses on "Setting up your own Package Cluster in MidnightBSD by Lucas Holt"

Leave a Message

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© HAKIN9 MEDIA SP. Z O.O. SP. K. 2013