Using Aptly for repository management

Direct publish⚓

It’s possible to publish repository directly with no snapshots. Unfortunately this is not possible with mirrors so if it’s needed to combine components including custom repository and mirrorred ones, it has to be via snapshots. See Multi component publishing

New publish

aptly publish repo -distribution=trusty-testing cloudlab

Update publish

aptly publish update trusty-testing

Snapshot publish⚓

To create new snapshot and create or update existing publish.

New publish

aptly snapshot create cloudlab-`date +%s` from repo cloudlab
 aptly publish snapshot -distribution=trusty-stable cloudlab-1433941328

Update publish

aptly snapshot create cloudlab-`date +%s` from repo cloudlab
 aptly publish switch -component=cloudlab trusty-stable cloudlab-1433941328

Sync mirrors⚓

It’s good idea to keep mirrors up to date and sync them periodically. You can use script bellow, place it eg. into /usr/local/bin/aptly_update_mirrors.sh and create crontab record for aptly user (suppose that aptly repo is not managed by root).

#!/bin/bash

SCRIPT=$(basename $0)
MAXTRIES=3
VERBOSE=0
SNAPSHOT=0

log_info() {
    logger -p user.info -t ${SCRIPT} "$*"
    [ $VERBOSE -eq 1 ] && echo "[INFO] $*"
}

log_error() {
    logger -p user.error -t ${SCRIPT} "$*"
    echo "[ERROR] $*" >&2
}

if [[ "$*" == *--verbose* || "$*" == *-v* ]]; then
    VERBOSE=1
fi

if [[ "$*" == *--snapshot* || "$*" == *-s* ]]; then
    SNAPSHOT=1
fi

MIRRORS=$(aptly mirror list --raw 2>&1)
if [[ $? -ne 0 ]]; then
    log_error "$MIRRORS"
    exit 1
fi

for mirror in $MIRRORS; do
    try=0
    retval=666

    while [[ $retval -ne 0 && $try -lt $MAXTRIES ]]; do
        log_info "Starting update of mirror ${mirror}"
        if [[ $VERBOSE -eq 0 ]]; then
            out=$(aptly mirror update -force=true ${mirror} >/dev/null 2>&1)
        else
            aptly mirror update -force=true ${mirror}
        fi

        retval=$?

        if [[ $retval -ne 0 ]]; then
            try=$[ $try + 1 ]
            log_error "Failed to update mirror ${mirror}, try=${try}"
            [ ! -z "$out" ] && log_error "${out}"
        else
            log_info "Synced mirror ${mirror}"
            if [[ $SNAPSHOT -eq 1 ]]; then
                snapshot_name="${mirror}-$(date +%s)"
                if [[ $VERBOSE -eq 0 ]]; then
                    out=$(aptly snapshot create ${snapshot_name} from mirror ${mirror} >/dev/null 2>&1)
                else
                    aptly snapshot create ${snapshot_name} from mirror ${mirror}
                fi

                retval=$?
                if [[ $retval -ne 0 ]]; then
                    log_error "Failed to create snapshot ${snapshot_name} from mirror ${mirror}"
                    [ ! -z "$out" ] && log_error "$out"
                else
                    log_info "Created snapshot ${snapshot_name} from mirror ${mirror}"
                fi
            fi
            break
        fi
    done
done

$ crontab -e -u aptly
# Sync aptly mirrors every 6 hours
0 */6 * * *  aptly_sync_mirrors.sh -v -s

First run may take a long time so it’s good idea to execute it in terminal multiplexer (Tmux, Screen, Byobu, etc.) in verbose mode. Parameter --snapshot will automatically create snapshots of newly synced mirrors.

aptly_sync_mirrors.sh --verbose --snapshot

Distribution⚓

We want to support only Ubuntu Trusty in three branches:

• nightly with up-to-date packages with mirrors
• testing for stabilization
• stable for rock-solid distribution

So final distributions will be:

• trusty-nightly
• trusty-testing
• trusty-stable

Components⚓

Each OS distribution has different approach to components. For our workflow, component means:

• set of independently versioned packages (sub-distribution) where customer may choose - eg. Openstack juno, kilo - or Opencontrail 21, 22, etc.
• set of packages with different or faster release workflow - because you can update publish per-component - eg. security from trusty-security distribution - or custom packages in our repository

For our example, components will be:

• main - merged upstream + ppa repositories with newer software versions
• security - security updates
  • because we want to push security updates between publishes separately
• cloudlab - our packages
  • because our packages can have different release cycle than rest of the distribution
• juno - packages of Openstack Juno
  • because this component is optional and one can use different openstack version than juno

Create publish⚓

This way you will create snapshots, merge some of them into one component snapshot (main) and publish using multi-component publishing.

Create snapshots for each mirror and repo

STAMP=$(date +%s)
aptly mirror list --raw|while read i;do aptly snapshot create $i-${STAMP} from mirror $i;done
aptly repo list --raw|while read i;do aptly snapshot create $i-${STAMP} from repo $i;done

Merge snapshots into desired final components

aptly snapshot merge main-${STAMP} salt-trusty-${STAMP} trusty-main-${STAMP}

Create multi-component publish of snapshots

aptly publish snapshot -component=main,cloudlab,juno,security -distribution=trusty-nightly main-${STAMP} cloudlab-${STAMP} trusty-updates-juno-${STAMP} trusty-security-${STAMP}

When new snapshots are created, use similar command as the one above to update publish:

aptly publish switch -component=main,cloudlab,juno,security trusty-nightly main-${STAMP} cloudlab-${STAMP} trusty-updates-juno-${STAMP} trusty-security-${STAMP}

For more information, read multi-component publishing

Mirrors⚓

There has to be one mirror per each repository distribution, so distribution has to be reflected in the name. Aptly doesn’t work with components on this level so each component has to be mirrored separately, otherwise it will be merged into main component.

Source repository can be ommited from mirror name if it’s upstream repository. Distribution name can me ommited if there’s only one distribution per mirrored repository and naming it explicitly would be confusing.

Patterns and examples

• <source>-<distribution>(-<component>)
• salt-trusty (<salt_ppa> trusty main)
• trusty-main (<ubuntu_upstream> trusty main)
• trusty-updates-juno (<ubuntucloud_repository> trusty-updates/juno main)

Snapshots⚓

It’s always good idea to use Unix timestamp for snapshot versioning and same name as for source mirror or repository.

For snapshots created by merging other snapshots, naming by published distribution is a good idea.

Patterns and examples

• cloudlab-$(date +%s)
• trusty-updates-juno-1434444631
• main-1434444631