Filip Pytloun — systems, agents & the work between

Distribution packages in Docker world

2016-08-16T17:22:19+02:00

..or why using them is still a good idea

Distribution packages

Software and it's dependencies maintained by Linux distribution vendor (Debian community, Canonical, RedHat, etc.), delivered by packages (DEB, RPM, ..) usually provided using package repositories (packages are signed so their authenticity is verified on download).

These packages actually forms stable Linux distribution because they are tested altogether. Especially Debian has great CI infrastructure where packaged software can be tested in case of dependency update.

As an example, let's take Python application which doesn't have much dependencies and generate dependency graph:

As you can see, you never deliver just application, you always deliver at least some dependencies, dependencies of dependencies and so on, including essential C libraries (eg. libssl in this example).

If anyone tells you that Docker is only about installing and running single application and base distribution doesn't matter.. don't belive it as he's only another naive utopist :-)

At the moment there's 860 packages in Debian repositories which directly depends on libssl and 404 packages which depends on python-requests.

And distribution maintainers and developers are here to ensure that update of libraries like python-requests or libssl won't break anything. Beside of that they are taking care of distribution purity, ensuring that every package has documentation, every binary has a man page, that copyright and license file is in place and that user understands what he is going to install. Again, especially Debian is very strict on this and has great set of pedantic tests to ensure everything is correct. Check resources at https://qa.debian.org, especially https://lintian.debian.org if you don't belive me :-)

Cons

Now to the bad things..

There is and will always be software which is not packaged for your distribution
- so yes, you need to teach people how to make packages
- and have some CI and repository management
Stable version of your distribution means older versions of software
- but you can run Debian testing or unstable which contains up-to-date software and this can be especially useful in Docker world where you can build image based on debian:unstable running just single up-to-date application
- or you can use your CI and repository management to make backports, that way you are basically creating your own Linux distribution
Distribution vendors also make mistakes
- so their minor update of single package can ruin your environment
- sometimes you need to fix something and don't wait for upstream
- so again, you need your own repository management and testing
Sometimes you need to run multiple versions (or series) of some software
- this can be done by repository components (eg. by OpenStack release - kilo, liberty, mitaka)
- or by prefix in package names (eg. python-, python3-)

Docker

First of all it's not true that container == Docker. Containers existed sooner than Docker (eg. LXC, OpenVZ) and was about running OS userspace (so using host's kernel) in very lightweight sort-of virtualization (far comparable with Xen paravirtualization).

Then came docker with tooling and essential thoughts about container management:

isolation done by facilities provided by Linux kernel - cgroups and namespaces, similar as LXC does
images built using Dockerfile, distributed using private Docker registry or public Dockerhub
container should run single application

This application focus brings many advantages over VM world:

complete isolation of individual pieces (applications)
much faster deployment (images are just ready to go)
easier rolling updates and management (scaling up/down, updating by spinning up new containers and just re-routing traffic and many new easy-to-be-done options)

The biggest deal (beside of orchestration of containers deployment which can be solved by tools like Kubernetes, Docker Swarm, etc.) which we are going to talk about is creation of Docker images.

Most Dockerfiles are similar to shell scripts, with structure like this:

use base debian/centos/whatever image
setup additional repositories if needed
install Python, Go or programming language of one's choice
install software using Pip or whatever management one's programming language offers
define entrypoint - application binary itself or shell script as a wrapper that will for example:
- generate configuration file using ENV variables it obtained on container launch (eg. mysql root password)
- do whatever preparations app needs to run (eg. create missing directories on volume)
- finally launch application

Does it remind you something? Déjà vu or maybe some scary memories of childhood? Didn't we already solve this once using configuration management.. SaltStack, Puppet, Ansible? Why are we going to make a step backward to shell scripts?

Fortunately it's possible to do this using your existing configuration management, see for example docker-salt project but back to our topic.

That way, developers can create docker images easily, in similar manner they are using for example Python's virtualenvs for local development. They can install anything, any version in any way, no restrictions. A dev's paradise. And they can upload final image so anyone can download and use it and it will work just in the same way.

But experienced sysadmin will surely ask some questions:

base image is comming from Dockerhub - who made that image, is it trustworthy? Is it cryptographically signed and signature verified on download?
core libraries (Python, dev libraries, possibly more) are comming from upstream distribution - how do I quickly patch all images in case of critical security issue in libssl?
application and it's dependencies are comming from Pypi - again, is it trustworthy source? Are all these packages signed and signature verified?
how can I be sure that critical security issue in dependency (eg. urllib) is fixed and developers just didn't blindly lock it's version using requirements.txt?
now we need to maintain multiple completely different ecosystems in case we are going to keep some "legacy" environments running VMs/bare-metal

And surely he will never deploy image from public dockerhub into production to avoid ending up like when Hitler uses Docker :-)

So it's clear there must be rules and defined workflows. Docker isn't package management tool - most people think they will just replace distribution packages by Docker but they will replace well-tested stable, security updated distribution packages by something similar to curl install.sh | sudo bash

But fortunately there's a solution - let Docker be revolutionary for developer use but take it evolutionary for production. Keep using distribution packages and just extend your workflows and repository management to cover build of docker images.

It will bring following advantages:

Repository management remains source of truth for versions going to be installed
- when it's snapshot or environment-based you can always return to stable set of packages, alter only versions you need (eg. security fixes)
- keeps still the same testing and QA process (on repository level), just extending it to use Docker images for infrastructure testing
Image builds are reproducable (per-environment repository locks versions) so you can make a small change in image without risk of being stucked on new version of upstream dependency, breaking up your app
By image creation date and snapshot of repository, you can easily tell which images needs update after critical security fix of package xyz
Build entirely from secure, trusted source (image build should have no Internet access except for package repository, which is rule that applies for package builds as well)

Repository management

So my advice is to have Repository management as a core component of all your workflows.

For Debian, you can use Aptly, for RHEL-based distributions there's Pulp which works in a very similar way.

Simple workflow can consist of 3 environments (but there can be more depending on company's use-cases).

nightly
- up-to-date mirrors of upstream and 3rd party APT repositories
- latest builds of your packages
- automated integration tests run against this environment
- nightly builds of docker images
testing
- promoted packages from nightly that passed automated testing
- base for staging environments where manual testing is done
- testing builds of docker images
stable
- source for production updates and build of production docker images
- QA approves promotion from testing
- stable builds of docker images

This workflow covers packages from all sources:

upstream distribution packages (eg. mirror of Debian repository)
3rd party repositories (various Launchpad PPAs, 3rd party vendors)
custom packages including:
- company's software
- additional packages
- backports and fixes of upstream packages

When using Aptly, you can automate these workflows using it's REST API. This can be done by re-using aptly-publisher which is CLI tool for management of Aptly publishes and designed for workflows similar to one described in these paragraphs.

This way, your private Docker registry will always reflect state of your packages repository. You can always deploy up-to-date myimage:stable or you can apt-get install myapp on VM or bare-metal and you will get the same result. You are still maintaining software in the same way for all ecosystems (no matter if it's container, Docker, VM or bare-metal) as you did before but wich much easier deployment and integration testing thanks to advantages of Docker.

Jabber anti-spam

2016-01-10T17:44:43+01:00

Introduction

There's no anti-spam mechanism in XMPP protocol so there's limited options one can do to avoid spam.

Server-side

Protect registration against automated spam account creation.

disable in-band registration completely
- and create accounts different way
or enable captcha during registration
- which may not be enough

There's currently no more servers can do to protect users from receiving spam.

Client-side

Only option is to block messages from users not in roster. This works pretty well but what matters is implementation:

use client which can block non-roster users by feature (eg. Psi)
- it's easy to setup, just one checkbox
- but this blocking will not work for other clients, eg. when using mobile client
use server-side privacy lists, XEP-0016

Setting privacy lists

Filtering using privacy lists is simple and works seamlessly for all clients.

This howto will describe how to setup privacy lists to block messages from contacts not in roster. Screenshots are from Gajim, but any client supporting XEP-0016 should be able to define these rules in a similar way.

Open Actions -> Advanced -> Edit Privacy Lists
Edit current privacy list (one should already exist), ensure it's active for this session and active on each startup.
Add rules with order 0 (rules are ordered from lower to higher) for each MUC domain, otherwise group chats will not work
Add deny rule with highest order (eg. 5). Only messages are blocked, subscribe requests are still allowed

Using LXC

2015-07-30T17:35:55+02:00

Intro

Containers are insulated areas inside a system, which have their own namespace for filesystem, network, PID, IPC, CPU and memory allocation and which can be created using the Control Group and Namespace features included in the Linux kernel.

Docker vs. LXC

The idea behind Docker is to reduce a container as much as possible to a single process and then manage that through Docker. This adds significant complexity. The main problem with this approach is you can't wish the OS away and the vast majority of apps and tools expect a multi process environment and support for things like cron, logging, ssh, daemons and with Docker since you have none of this you have do everything via Docker from basic app configuration to deployment, networking, storage and orchestration.

LXC sidesteps that with a normal OS environment and is thus immediately and cleanly compatible all the apps and tools and any management and orchestration layers and be a drop in replacement for VMs.

Beyond that Docker uses layers and disables storage persistence. LXC supports layers with aufs, overlayfs and has wide support for COW cloning and snapshots with btrfs, ZFS, LVM Thin and leaves their use of these to user choice. Separating storage is a simple bind mount to the host or another container for users who choose to do so.

-- source: flockport.com

Install lxc

apt-get install lxc

Sudo

To be able to execute lxc commands as root without password, you need to configure sudo.

echo "$USER  ALL=(root) NOPASSWD: /usr/bin/lxc-*" >> /etc/sudoers.d/${USER}

Setup network

NAT

Probably best option when you are using laptop and moving between various networks.

First install bridge-utils and DHCP server (optional)

apt-get install bridge-utils isc-dhcp-server

Configure /etc/network/interfaces

auto br0
iface br0 inet static
   address 172.20.0.1
   netmask 255.255.255.0
   bridge_stp off
   bridge_maxwait 5
   pre-up  /sbin/brctl addbr br0
   post-up /sbin/brctl setfd br0 0
   post-down /sbin/brctl delbr br0
   post-up /sbin/iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
   post-up /sbin/iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT
   post-up /sbin/iptables -A FORWARD -i wlan0 -o br0 -j ACCEPT
   post-down /sbin/iptables -t nat -D POSTROUTING -o wlan0 -j MASQUERADE
   post-down /sbin/iptables -D FORWARD -i br0 -o wlan0 -j ACCEPT
   post-down /sbin/iptables -D FORWARD -i wlan0 -o br0 -j ACCEPT

That was a little bit longer, replace wlan0 by your network primary interface. Address and netmask defines your new network. Choose something that won't be in collision with real networks that you use.

You also need to enable IP forwarding, ensure you have following line in /etc/sysctl.conf:

net.ipv4.ip_forward=1

And reload sysctl

sysctl -p

Bring up network interface

ifup br0

You also need to setup DHCP server so your containers will get IP address automatically.

Edit /etc/default/isc-dhcp-server and set interfaces to listen on

INTERFACES="br0"

Edit /etc/dhcp/dhcpd.conf and setup your subnet

subnet 172.20.0.0 netmask 255.255.255.0 {
    range 172.20.0.10 172.20.0.100;
    option routers 172.20.0.1;
    option domain-name "lxc.hydra";
    option domain-name-servers 8.8.4.4, 8.8.8.8;
}

Finally start dhcpd

service isc-dhcp-server restart

And last step, edit /etc/lxc/default.conf

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

Create container

Debian

sudo lxc-create -n my_container -t debian

Ubuntu

To bootstrap Ubuntu container on Debian, you need to install ubuntu keyring first.

apt-get install ubuntu-archive-keyring

sudo lxc-create -n ubuntu_container -t ubuntu

Container management

List containers

sudo lxc-ls -f

Get container informations

sudo lxc-info --name my_container

Start container (add -d parameter to run as daemon)

sudo lxc-start --name my_container

Stop running container

sudo lxc-stop --name my_container

Destroy container

sudo lxc-destroy --name my_container

Enter root shell within container

sudo lxc-attach --name ubuntu
# Or enter console
sudo lxc-console --name ubuntu

Execute command inside container

sudo lxc-attach --name ubuntu -- hostname

Advanced usage

Create container on LVM (without --lvname LV name will be same as container)

lxc-create -n debian -t debian -B lvm --vgname vg0 --lvname lxc_debian

Reference

Using libvirt and KVM

2015-07-27T17:35:55+02:00

Installation

Install required packages

apt-get install qemu-kvm libvirt-bin bridge-utils virtinst

Setup /etc/network/interfaces

iface eth0 inet manual

auto br0
iface br0 inet static
    bridge_interfaces eth0
    bridge_stp off
    bridge_maxwait 5
    .. address, netmask, etc.

Try to put br0 interface up or reboot

brctl addbr br0

Deploy instance

Now you have more options

deploy from custom image
deploy clean Ubuntu using ubuntu-vm-builder

Ubuntu VM builder

Attention!

Untested

apt-get install ubuntu-vm-builder

ubuntu-vm-builder kvm trusty --rootsize=10000 --domain cfg01 --dest cfg01
--hostname cfg01 --arch amd64 --mem 4096 --ip 172.19.5.30 --mask 255.255.255.0
--net 172.19.0.0 --gw 172.19.5.0 --dns 172.19.0.1 --libvirt qemu:///system
--addpkg openssh-server --addpkg acpid --addpkg vim-nox --components
"main,universe"

Boot from image

apt-get install virtinst

Get image and prepare image for guest (you can use overlay image instead)

cd /var/lib/libvirt/images
wget http://apt.cloudlab.cz/images/ubuntu-14-04-x64-1437659147.qcow2
cp ubuntu-14-04-x64-1437659147.qcow2 cfg01.qcow2

Install guest

virt-install --name cfg01 \
    --ram 4096 --vcpus=2 \
    --accelerate --network bridge:br0 \
    --disk path=/var/lib/libvirt/images/cfg01.qcow2,bus=virtio,cache=none \
    --boot hd --vnc --noreboot --autostart

Using overlay image

To save disk space, you can use overlay image instead of copying original image for each instance.

qemu-img create -f qcow2 -b ubuntu-14-04-x64-1437659147.qcow2 cfg01.ovl

Then simply use *.ovl for --disk parameter of virt-install command.

Attention!

Don't ever touch original image

Cloud-init configuration

To use cloud-init with libvirt, you need to provide meta-data and user-data somehow. There are multiple data sources that cloud-init expects, the simplest way is probably to generate configuration iso and attach it to instance cdrom.

Install dependencies

apt-get install genisoimage
curl -q https://raw.githubusercontent.com/larsks/virt-utils/master/create-config-drive | sed s,/bin/sh,/bin/bash,g > create-config-drive
chmod +x create-config-drive

Prepare userdata file, eg. userdata.sh

#!/bin/bash

sed -i '/iface\ eth0\ inet\ dhcp/d' /etc/network/interfaces
cat << EOF >> /etc/network/interfaces
iface eth0 inet static
  address 172.19.5.101
  netmask 255.255.255.0
  gateway 172.19.5.254
  dns-nameservers 172.19.0.1
EOF

ifdown eth0; ifup eth0

echo "id: test1.int.cloudlab.cz" >> /etc/salt/minion.d/minion.conf
echo "master: 172.19.5.30" >> /etc/salt/minion.d/minion.conf
service salt-minion restart

salt-call state.highstate

Generate configuration iso image

./create-config-drive -k ~/.ssh/id_rsa.pub -u userdata.sh -h test1 test1-config.iso

Finally you can create domain

virt-install --name test1 \
    --ram 4096 --vcpus=2 \
    --accelerate --network bridge:br0 \
    --disk path=/var/lib/libvirt/images/test1.qcow2,bus=virtio,cache=none
    --disk path=test1-config.iso,device=cdrom \
    --boot hd --vnc --noreboot --autostart \

Manage instance

List instances

virsh list --all

Stop instance gracefuly

virsh shutdown cfg01

Stop instance by force in case it's hanged

virsh destroy cfg01

Delete instance

virsh undefine cfg01

Resize disk

Attention!

Shut down instance first, this is important

You can resize both overlay image or normal image. But don't ever resize image that is source for overlay images.

qemu-img resize cfg01.ovl +10G

On reboot, cloud-init should automatically grow root filesystem.

Extend memory

First you need to redefine max memory for guest:

virsh setmaxmem cfg01 16G --config

After that you need to destroy domain and start it again.

Finally you can set memory onlin with following command:

virsh setmem cfg01 16G --config --live

Parameter --config will update domain XML so changes will preserve boot.

Resources

Launchpad tricks

2015-07-20T17:33:37+02:00

Upload issues

File `*.orig.tar.gz` already exists

File python-redis_2.10.1.orig.tar.gz already exists in Primary Archive for Ubuntu, but uploaded version has different contents.

You cannot upload file that already exists in Launchpad but has different content (file checksum). File checksum changes every time you re-create gzipped tarball, because it modifies gzip headers.

This happens for example if you import original source package from Ubuntu archive, make some fixes, eg. backport fixes, create your own source package by gbp buildpackage -S and want to upload it to PPA.

Fortunately the "fix" is simple - you need to download original tarball from Ubuntu archive and use it in your source package.

You can use script similar to this one to download original tarball from archive and replace checksums in your *.dsc file.

#!/bin/bash -x

PPA_REPO="fpytloun/extra"

orig=$(ls *.orig.tar.gz)
orig_size=$(ls -l *.orig.tar.gz | cut -d ' ' -f 5)
orig_sha1=$(sha1sum $orig | cut -d ' ' -f 1)
orig_sha256=$(sha256sum $orig | cut -d ' ' -f 1)
orig_md5=$(md5sum $orig | cut -d ' ' -f 1)

wget --quiet -O $orig-tmp "https://launchpad.net/ubuntu/+archive/primary/+files/$orig"
if [ $? -eq 0 ]; then
    echo "[WARN] Original tarball found in Ubuntu archive, using it instead"
    mv $orig-tmp $orig
    new_size=$(ls -l *.orig.tar.gz | cut -d ' ' -f 5)
    new_sha1=$(sha1sum $orig | cut -d ' ' -f 1)
    new_sha256=$(sha256sum $orig | cut -d ' ' -f 1)
    new_md5=$(md5sum $orig | cut -d ' ' -f 1)
    sed -i -e s,$orig_sha1,$new_sha1,g -e s,$orig_sha256,$new_sha256,g -e s,$orig_size,$new_size,g -e s,$orig_md5,$new_md5,g *.dsc
    sed -i -e s,$orig_sha1,$new_sha1,g -e s,$orig_sha256,$new_sha256,g -e s,$orig_size,$new_size,g -e s,$orig_md5,$new_md5,g *.changes
fi

set -e
debsign --re-sign *.changes
dput -f "ppa:$PPA_REPO" *.changes

Alternatively you can re-build source package using original tarball so you don't need to fix checksums manually.

Building OpenContrail

2015-06-22T12:29:00+02:00

Dependencies

Build dependencies

Dependencies needed to build Opencontrail.

Actual deps in tools/packages/debian/contrail/debian/builddep.trusty

cat tools/packages/debian/contrail/debian/builddep.trusty|sed s,[\ \\\t],,g|tr -d ','|tr '\n' ' '

libboost-dev libboost-chrono-dev libboost-date-time-dev
libboost-filesystem-dev libboost-program-options-dev libboost-python-dev
libboost-regex-dev libboost-system-dev libboost-thread-dev
libcurl4-openssl-dev google-mock libgoogle-perftools-dev liblog4cplus-dev
libtbb-dev libhttp-parser-dev libxml2-dev libicu-dev

Dependencies for building 3rd party packages:

python-lxml autoconf automake bzip2 libtool patch unzip wget

And a few more, see Build-Depends: section of tools/packages/debian/contrail/debian/control

scons git python-lxml wget gcc patch make unzip flex bison g++ libssl-dev
autoconf automake libtool pkg-config vim python-dev python-setuptools
libprotobuf-dev protobuf-compiler libsnmp-python librdkafka-dev

Note that you need latest librdkafka (version 0.8.6) to provide librdkafka++.so It is not present in Debian or Ubuntu repositories so you need to find it somewhere on Launchpad or build on your own.

You also need libipfix. It's not in Debian repository so you need to build it on your own or grab it here: https://launchpad.net/~opencontrail/+archive/ubuntu/ppa/+packages

Some more dependencies may be needed, I have found out at least following:

debhelper libxml2-utils python-all python-sphinx ruby-ronn
module-assistant ant default-jdk javahelper libcommons-codec-java
libhttpcore-java liblog4j1.2-java

Cowbuilder

To build debian packages with Cowbuilder/Pbuilder, it will handle dependencies by itself in chrooted environment.

Therefore you need just a few packages to be able to build debian source files.

libxml2-utils
python-lxml
autoconf
automake
libtool
patch
unzip
pkg-config
javahelper
ant
python-setuptools

Install dependencies

To install opencontrail packages, you need to satisfy some dependencies that are not in Ubuntu Trusty installation.

python-geventhttpclient (>= 1.0)
python-certifi
python-backports.ssl-match-hostname
python-ncclient
python-pycassa (>= 1.7.2)
python-xmltodict
python-docker-py (python-opencontrail-vrouter-netns)
python-redis (>= 2.8.0)

Build

Manual

Follow these steps to build Opencontrail manually or skip to building with Jenkins and Cowbuilder.

Get android repo tool

wget https://storage.googleapis.com/git-repo-downloads/repo
chmod +x repo

Initialize sandbox

mkdir build;cd build
repo init -u git@github.com:Juniper/contrail-vnc.git -b ${CONTRAIL_BRANCH:-master}
repo sync

Get and patch 3rd party packages

cd third_party
./fetch_packages.py

Execute the build

# For building into dest root, no debian package
scons -j ${JOBS_COUNT:-$(grep -c processor /proc/cpuinfo || echo 1)}

# Or create Debian packages, they will appear in build/packages/ directory
make -f packages.make all

# You can also build only source packages and build binaries with
# cowbuilder or some better way that will also handle dependencies
make -f packages.make source-all

Jenkins

To automate building with Jenkins, you can use script like this:

#!/bin/bash -xe

export USER=$(whoami)
JOBS_COUNT=${JOBS_COUNT:-$(cat /proc/cpuinfo | grep -c processor || echo 1)}

# Github account is needed to fetch repositories
pgrep -l -u $USER -f | grep -e ssh-agent\$ >/dev/null || ssh-agent|grep -v "Agent pid" > ~/.ssh/ssh-agent.sh
. ~/.ssh/ssh-agent.sh
ssh-add ${SSH_KEY:-"$HOME/.ssh/id_rsa"}
grep github ~/.ssh/known_hosts || ssh-keyscan github.com >> ~/.ssh/known_hosts

if [ ! -d bin ]; then
    mkdir bin
    wget -O bin/repo https://storage.googleapis.com/git-repo-downloads/repo
    chmod +x bin/repo
fi

PATH="$WORKSPACE/bin:$PATH"

[ ! -d build ] && mkdir build
cd build
if [[ "$CONTRAIL_BRANCH" == "default" ]]; then
    repo init -u $CONTRAIL_VNC_REPO
else
    repo init -u $CONTRAIL_VNC_REPO -b $CONTRAIL_BRANCH
fi
repo sync

cd third_party
python fetch_packages.py
cd ..

# Normal build
# scons -j $JOBS_COUNT
# Build debian packages
#  make -f packages.make all
# Build debian source packages for binary build
# unfortunately source-all target doesn't work
#  make -f packages.make source-all
grep '^source-package-.*:' packages.make | cut -d : -f 1 | while read i; do
    make -f packages.make $i
done

You could possibly also use Jenkins repo plugin for SCM management.

Cowbuilder

If you are going to build debian packages, I would advice to use make -f packages.make source-all in the script above.

This will create *.dsc files that can be used as source artifacts for build with Cowbuilder which can create build chroot and handle build dependencies without garbaging your system.

Binary build job can look like this, using jenkins-debian-glue:

#!/bin/bash -xe
for i in *.dsc; do
    pkgname=$(echo $i|cut -d "_" -f 1)
    [ ! -d $pkgname ] && mkdir $pkgname
    mv ${pkgname}_*.gz ${pkgname}_*.dsc ${pkgname}/
    cd $pkgname
    export WORKSPACE=$(pwd)
    /usr/bin/build-and-provide-package
    cd ..
done
export WORKSPACE=$(pwd)

Don't forget you need to install additional dependencies that are not present in Debian or Ubuntu repositories.

To setup additional repository, update /etc/pbuilderrc:

OTHERMIRROR="deb http://apt.tcpcloud.eu/nightly cloudlab"

Or to use your only own repository for both debootstrap and build, you can use setup similar to this:

MIRRORSITE="http://apt.tcpcloud.eu/nightly"
COMPONENTS="main cloudlab security"
# Debootstrap in Ubuntu is using ubuntu keyring, to use # your own, provide
# default keyring
DEBOOTSTRAPOPTS=(${DEBOOTSTRAPOPTS[@]} "--keyring=/etc/apt/trusted.gpg")
APTKEYRINGS=(${APTKEYRINGS[@]} "/etc/apt/trusted.gpg")

Unfortunately if you have different list of components than are default for your distribution, you need to export COMPONENTS before running build-and-provide-package otherwise jenkins debian glue will override these provided in pbuilderrc.

References

Using Aptly for repository management

2015-06-15T20:30:35+02:00

Repository anatomy

Record in sources.list may look like this:

deb http://ppa.launchpad.net/saltstack/salt/ubuntu trusty main

It consists of repository url, distribution and one or more components.

deb <repository> <distribution> <component(s)>

Distribution

Distribution name consists of Linux distribution version codename (eg. trusty) and our internal stability identifier (stable or testing)

It can also have a prefix that can identify source.

Examples

trusty-stable
trusty-testing
upstream/trusty-testing

Component

Components are subsets of each distribution repository. They can be used independently by choice or all together.

Examples

main - Ubuntu packages
universe - Ubuntu universe packages

Local repository

Custom packages build by us. Published as cloudlab component.

aptly repo create -component=cloudlab -architectures=amd64 cloudlab

Publishing

Direct publish

It's possible to publish repository directly with no snapshots. Unfortunately this is not possible with mirrors so if it's needed to combine components including custom repository and mirrorred ones, it has to be via snapshots. See Multi component publishing

New publish

aptly publish repo -distribution=trusty-testing cloudlab

Update publish

aptly publish update trusty-testing

Snapshot publish

To create new snapshot and create or update existing publish.

New publish

aptly snapshot create cloudlab-`date +%s` from repo cloudlab
 aptly publish snapshot -distribution=trusty-stable cloudlab-1433941328

Update publish

aptly snapshot create cloudlab-`date +%s` from repo cloudlab
 aptly publish switch -component=cloudlab trusty-stable cloudlab-1433941328

Repository mirror

First you need to import distributor's GPG key to be able to verify packages.

gpg --no-tty --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 0E27C0A6

Create mirror

aptly mirror create -architectures=amd64 salt http://ppa.launchpad.net/saltstack/salt/ubuntu/ trusty main

Update mirror

This will fetch newest packages from upstream. Should be run periodically to be always up to date.

aptly mirror update salt

Real-world example

Let's try to design some real-world example.

Software sources

local repository with custom packages
upstream mirror
- trusty main
- trusty-security main
- ubuntucloud (trusty-updates/juno main)
salt repository mirror

Example status

aptly@aptly01:/home/filip$ aptly repo list
List of local repos:
 - [cloudlab]: TCP Cloud custom components (packages: 0)

aptly@aptly01:/home/filip$ aptly mirror list
List of mirrors:
 - [salt-trusty]: http://ppa.launchpad.net/saltstack/salt/ubuntu/ trusty
 - [trusty-main]: http://eu.archive.ubuntu.com/ubuntu/ trusty
 - [trusty-security]: http://eu.archive.ubuntu.com/ubuntu/ trusty-security
 - [trusty-updates-juno]: http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/juno

Sync mirrors

It's good idea to keep mirrors up to date and sync them periodically. You can use script bellow, place it eg. into /usr/local/bin/aptly_update_mirrors.sh and create crontab record for aptly user (suppose that aptly repo is not managed by root).

#!/bin/bash

SCRIPT=$(basename $0)
MAXTRIES=3
VERBOSE=0
SNAPSHOT=0

log_info() {
    logger -p user.info -t ${SCRIPT} "$*"
    [ $VERBOSE -eq 1 ] && echo "[INFO] $*"
}

log_error() {
    logger -p user.error -t ${SCRIPT} "$*"
    echo "[ERROR] $*" >&2
}

if [[ "$*" == *--verbose* || "$*" == *-v* ]]; then
    VERBOSE=1
fi

if [[ "$*" == *--snapshot* || "$*" == *-s* ]]; then
    SNAPSHOT=1
fi

MIRRORS=$(aptly mirror list --raw 2>&1)
if [[ $? -ne 0 ]]; then
    log_error "$MIRRORS"
    exit 1
fi

for mirror in $MIRRORS; do
    try=0
    retval=666

    while [[ $retval -ne 0 && $try -lt $MAXTRIES ]]; do
        log_info "Starting update of mirror ${mirror}"
        if [[ $VERBOSE -eq 0 ]]; then
            out=$(aptly mirror update -force=true ${mirror} >/dev/null 2>&1)
        else
            aptly mirror update -force=true ${mirror}
        fi

        retval=$?

        if [[ $retval -ne 0 ]]; then
            try=$[ $try + 1 ]
            log_error "Failed to update mirror ${mirror}, try=${try}"
            [ ! -z "$out" ] && log_error "${out}"
        else
            log_info "Synced mirror ${mirror}"
            if [[ $SNAPSHOT -eq 1 ]]; then
                snapshot_name="${mirror}-$(date +%s)"
                if [[ $VERBOSE -eq 0 ]]; then
                    out=$(aptly snapshot create ${snapshot_name} from mirror ${mirror} >/dev/null 2>&1)
                else
                    aptly snapshot create ${snapshot_name} from mirror ${mirror}
                fi

                retval=$?
                if [[ $retval -ne 0 ]]; then
                    log_error "Failed to create snapshot ${snapshot_name} from mirror ${mirror}"
                    [ ! -z "$out" ] && log_error "$out"
                else
                    log_info "Created snapshot ${snapshot_name} from mirror ${mirror}"
                fi
            fi
            break
        fi
    done
done

$ crontab -e -u aptly
# Sync aptly mirrors every 6 hours
0 */6 * * *  aptly_sync_mirrors.sh -v -s

First run may take a long time so it's good idea to execute it in terminal multiplexer (Tmux, Screen, Byobu, etc.) in verbose mode. Parameter --snapshot will automatically create snapshots of newly synced mirrors.

aptly_sync_mirrors.sh --verbose --snapshot

Publishing

First we need to decide how our published repository will look like.

Distribution

We want to support only Ubuntu Trusty in three branches:

nightly with up-to-date packages with mirrors
testing for stabilization
stable for rock-solid distribution

So final distributions will be:

trusty-nightly
trusty-testing
trusty-stable

Components

Each OS distribution has different approach to components. For our workflow, component means:

set of independently versioned packages (sub-distribution) where customer may choose - eg. Openstack juno, kilo - or Opencontrail 21, 22, etc.
set of packages with different or faster release workflow - because you can update publish per-component - eg. security from trusty-security distribution - or custom packages in our repository

For our example, components will be:

main - merged upstream + ppa repositories with newer software versions
security - security updates
- because we want to push security updates between publishes separately
cloudlab - our packages
- because our packages can have different release cycle than rest of the distribution
juno - packages of Openstack Juno
- because this component is optional and one can use different openstack version than juno

Create publish

This way you will create snapshots, merge some of them into one component snapshot (main) and publish using multi-component publishing.

Create snapshots for each mirror and repo

STAMP=$(date +%s)
aptly mirror list --raw|while read i;do aptly snapshot create $i-${STAMP} from mirror $i;done
aptly repo list --raw|while read i;do aptly snapshot create $i-${STAMP} from repo $i;done

Merge snapshots into desired final components

aptly snapshot merge main-${STAMP} salt-trusty-${STAMP} trusty-main-${STAMP}

Create multi-component publish of snapshots

aptly publish snapshot -component=main,cloudlab,juno,security -distribution=trusty-nightly main-${STAMP} cloudlab-${STAMP} trusty-updates-juno-${STAMP} trusty-security-${STAMP}

When new snapshots are created, use similar command as the one above to update publish:

aptly publish switch -component=main,cloudlab,juno,security trusty-nightly main-${STAMP} cloudlab-${STAMP} trusty-updates-juno-${STAMP} trusty-security-${STAMP}

For more information, read multi-component publishing

Naming conventions

Mirrors

There has to be one mirror per each repository distribution, so distribution has to be reflected in the name. Aptly doesn't work with components on this level so each component has to be mirrored separately, otherwise it will be merged into main component.

Source repository can be ommited from mirror name if it's upstream repository. Distribution name can me ommited if there's only one distribution per mirrored repository and naming it explicitly would be confusing.

Patterns and examples

<source>-<distribution>(-<component>)
salt-trusty (<salt_ppa> trusty main)
trusty-main (<ubuntu_upstream> trusty main)
trusty-updates-juno (<ubuntucloud_repository> trusty-updates/juno main)

Snapshots

It's always good idea to use Unix timestamp for snapshot versioning and same name as for source mirror or repository.

For snapshots created by merging other snapshots, naming by published distribution is a good idea.

Patterns and examples

cloudlab-$(date +%s)
trusty-updates-juno-1434444631
main-1434444631

Filip Pytloun — systems, agents & the work between

Distribution packages in Docker world

..or why using them is still a good idea

Distribution packages

Cons

Docker

Repository management

Jabber anti-spam

Introduction

Server-side

Client-side

Setting privacy lists

Using LXC

Intro

Docker vs. LXC

Install lxc

Sudo

Setup network

NAT

Create container

Debian

Ubuntu

Container management

Advanced usage

Reference

Using libvirt and KVM

Installation

Deploy instance

Ubuntu VM builder

Boot from image

Using overlay image

Cloud-init configuration

Manage instance

Resize disk

Extend memory

Resources

Launchpad tricks

Upload issues

File *.orig.tar.gz already exists

Building OpenContrail

Dependencies

Build dependencies

Cowbuilder

Install dependencies

Build

Manual

Jenkins

Cowbuilder

References

Using Aptly for repository management

Repository anatomy

Distribution

Component

Local repository

Publishing

Direct publish

Snapshot publish

Repository mirror

Real-world example

Software sources

Sync mirrors

Publishing

Distribution

Components

Create publish

Naming conventions

Mirrors

Snapshots

File `*.orig.tar.gz` already exists