Steve's Solaris/Linux experiences: puppet, git, gitlab & r10k: Split that Monolithic git workflow

Configuration Management and Version Control Software! Definitely, a great way to start a new year in this DevOps' era is to write a post about these two hot topics. Indeed, Moving forward on my DevOPS path makes these two unavoidable.

I'm using Puppet for more than a year now, I have written a dozen of useful modules and as I'm introducing the rest of the team to Configuration Management/Puppet, I'm also starting to understand the great benefits of using Version Control Software(VCS) to manage Puppet Code (or any other Source Codes), especially when that code is being edited by many people (it's even more obvious when these other people are SysAdmin).

The aim of this post is mainly to share my experience about moving from that monolithic environment where I used to edit my Written Puppet Modules directly on my Puppet Master and use git (a bunch of git clone/git remote/git push/git fetch...) for maintaining Puppet Directory Environment (Production/Dev/Test Puppet Environments) to a much cleaner Workflow where I (and any others contributors) can easily contribute. On that cleaner workflow, I'm intending to make use of:

A smart tool which provides a general purpose toolset for deploying Puppet environments and modules in a seamless way (R10k).

An Internal Online Repository Hosting System (Gitlab CE/Git) to Publish /Store Puppet Codes in a Central Repo.

A much modern integrated toolset for developing Puppet modules and manifests (Gepetto) from workstation (not from my Puppet Master).

Note that similar workflows have already been discussed on many great websites (some are listed in the references sections), but I'm intending to share (with as much details as possible) my own Experience moving from a monolithic workflow to one that has the set of features that are listed above, My hope is that this will help others people designing their workflow.

To make it more readable, I've divided this post in 03 sections:

I. A Basic and Monolithic Git workflow: Description of this Monolithic Environment

This section is about describing in details the Basic Monolithic Git workflow and how that was implemented. I will also highlight some pros and cons of that workflow.

II. Moving to "one repo per Puppet Module" and a Centralized Git

In This section, I'm describing in detail the split of the huge Puppet Repositories in many smaller repo that are hosted on a Centralized Enterprise Git Server (GitLab-CE)

III. Plastering the walls with r10k:

This last section is about r10k Implementation for the synchronization of the Puppet Environments, their associate Modules and Data

I. A Basic and Monolithic Git workflow : Description of the Monolithic Environment

*** One ring to rule them all

This section is about describing in details the Basic Monolithic Git workflow and how that was implemented.

The Setup I'm using for the post is a classic one for a Middle-sized Infrastructure, a PE 2015.2 with Puppet master, PE console, and PuppetDB all installed on one node.

For Puppet Directory Environment (production/Dev/Test...), I'm using the recommended PE 2015.2 environment-based workflow for testing new code in the node classifier. this workflow is well described on Puppet Site, but basically, it's about the creation/configuration of two type of Nodes Groups, a first set of node groups is used exclusively for assigning environments to nodes, and the second set of node groups is used for applying classes to nodes.

Below, a Picture which summarizes that workflow. On the left side, we have a set of node groups which are used for assigning nodes to either Production, Dev or Test environments and on the right side, the structure that is put in place for Classifying Nodes (using roles).

Source: http://docs.puppetlabs.com/pe/2015.2/console_classes_groups_environment_override.html#

For the implementation of such design, I've the default production environment and new test and Dev Environments. The test and dev environments were simply created by initializing /etc/puppetlabs/code/environments/production as git repo and cloning that repo to both dev test directories . The Synchronization between test/dev and production repo is (obviously) manual, with the development of new features/modules being done exclusively in the test environment. Once satisfied with the features/modules on tests environment, the test repository is merged in dev environment and later on, to Production environment. Below is a picture that try to summarize this basic git workflow.

I.a: Creation of Git Repo and Directory Environments

Let's see how this is implemented on my Puppet Master (Name: pe-master). Note that I'm using the default (already configured) production directory to initialize the First git repo.

### Creation of production repo
[root@pe-master production]# cd /etc/puppetlabs/code/environments/production
[root@pe-master production]# git init 
Initialized empty Git repository in /etc/puppetlabs/code/environments/production/.git/

[root@pe-master production (master)]# git add --all :/
[root@pe-master production (master)]# git commit -m "Production Initial Repository with few official modules installed"
[......]

#### Cloning of production repo to test and dev Repos
[root@pe-master production]# cd ..
[root@pe-master environments]# git clone production test
Cloning into 'test'...
done.
[root@pe-master environments]# git clone production dev
Cloning into 'dev'...
done.

### Adding Remote repositories to test/dev Repositories
[root@pe-master environments]# cd test/
[root@pe-master  test (master)]# pwd
/etc/puppetlabs/code/environments/test
[root@pe-master test (master)]# git remote add dev /etc/puppetlabs/code/environments/dev
[root@pe-master test (master)]$ git fetch dev 
From /etc/puppetlabs/code/environments/dev
 * [new branch]      master     -> dev/master
[root@pe-master test (master)]# git remote -v
dev     /etc/puppetlabs/code/environments/dev (fetch)
dev     /etc/puppetlabs/code/environments/dev (push)
origin  /etc/puppetlabs/code/environments/production (fetch)
origin  /etc/puppetlabs/code/environments/production (push)
[root@pe-master test (master)]# git branch -r
  dev/master
  origin/HEAD -> origin/master
  origin/master
[root@pe-master test (master)]# cd ../dev
[root@pe-master dev (master)]# pwd
/etc/puppetlabs/code/environments/dev
[root@pe-master test]# git status
# On branch master
nothing to commit, working directory clean
[root@pe-master dev (master)]# git remote add test /etc/puppetlabs/code/environments/test
[root@pe-master dev (master)]$ git fetch test 
From /etc/puppetlabs/code/environments/test
 * [new branch]      master     -> test/master
[root@pe-master dev (master)]# git remote -v
origin  /etc/puppetlabs/code/environments/production (fetch)
origin  /etc/puppetlabs/code/environments/production (push)
test    /etc/puppetlabs/code/environments/test (fetch)
test    /etc/puppetlabs/code/environments/test (push)
[root@pe-master dev (master)]# git branch -r
  origin/HEAD -> origin/master
  origin/master
  test/master

### Checking the Production Repositories and adding test/dev as remote
[root@pe-master dev (master)]# cd ../production
[root@pe-master production (master)]$ git remote add test /etc/puppetlabs/code/environments/test    
[root@pe-master production (master)]$ git remote add dev /etc/puppetlabs/code/environments/dev 
[root@pe-master production (master)]$ git fetch test 
From /etc/puppetlabs/code/environments/test
 * [new branch]      master     -> test/master
[root@pe-master production (master)]$ git fetch dev
From /etc/puppetlabs/code/environments/dev
 * [new branch]      master     -> dev/master
[root@pe-master production (master)]# git remote -v
dev     /etc/puppetlabs/code/environments/dev (fetch)
dev     /etc/puppetlabs/code/environments/dev (push)
test    /etc/puppetlabs/code/environments/test (fetch)
test    /etc/puppetlabs/code/environments/test (push)

I.b: Creation of Environments Nodes Groups:

As test and dev Environment Directories are already created, the node classification under these Environments Node Groups (test/dev) is quite easy on Puppet Enterprise Console.

On the enterprise console, under the node ==> classification section, we create a new group named test (dev) and set the Environment to "test".

Once that is done, we go on that group and "Edit node group metadata", then checked " This is an environment group button

I.c: Create and use a new Module:

With this implementation, Modules can be written in the test environment and added in a seamless way to the dev environment and later on to production environment by just merging the git repositories.
Let's illustrate this by creating a quick (and dirty) newfeature module ( which simply notify on the environment facts).

### Get to the test modules directory and create newfeature Branch

[root@pe-master test (master)]# cd modules/
[root@pe-master modules (master)]# git status 
# On branch master
nothing to commit, working directory clean
[root@pe-master modules (master)]# git checkout -b newfeature
Switched to a new branch 'newfeature'
[root@pe-master modules (newfeature)]# git status
# On branch newfeature
nothing to commit, working directory clean
[root@pe-master modules (newfeature)]# pwd
/etc/puppetlabs/code/environments/test/modules
[root@pe-master modules (newfeature)]# git branch 
  master
* newfeature



### Edit Puppet Manifest

[root@pe-master modules (newfeature)]# mkdir -p newfeature/{manifests,tests,files}
[root@pe-master modules (newfeature)]# cd newfeature/manifests/
[root@pe-master manifests (newfeature)]# ls
[root@pe-master manifests (newfeature)]# vim init.pp
[root@pe-master manifests (newfeature)]# cat init.pp 
class newfeature {
        notify { "This is the environment I am assigned to : ${::environment}": }
}
[root@pe-master manifests (newfeature)]# puppet parser validate init.pp 
[root@pe-master manifests (newfeature)]# puppet module list --environment test | grep newfeature

├── newfeature (???)

### Commit this change

[root@pe-master manifests (newfeature)]# cd /etc/puppetlabs/code/environments/test/modules
[root@pe-master modules (newfeature)]# git status 
# On branch newfeature
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       newfeature/
nothing added to commit but untracked files present (use "git add" to track)
[root@pe-master modules (newfeature)]# git add newfeature
[root@pe-master modules (newfeature)]# git commit -m "Added newfeature module"
[newfeature f037c7b] Added newfeature module
 Committer: root <root@pe-master.stiv.local>
 1 file changed, 3 insertions(+)
 create mode 100644 modules/newfeature/manifests/init.pp

Classify a test node (Add my small newfeature to the right profile) and run puppet agent on that test node.

[root@pe-test ~]# puppet agent -t
Warning: Local environment: "production" doesn't match server specified node environment "test", switching agent to "test".
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for pe-test.stiv.local
Info: Applying configuration version '1452445799'
Notice: This is the environment I am assigned to : test
Notice: /Stage[main]/newfeature/Notify[This is the environment I am assigned to : test]/message: defined 'message' as 'This is the environment I am assigned to : test'
Notice: Applied catalog in 0.73 seconds

The test seems to be well working, so we can merge it back to our test master branch and delete the newfeature branch.

[root@pe-master modules (newfeature)]# git branch 
  master
* newfeature
[root@pe-master modules (newfeature)]# git checkout master
Switched to branch 'master'
[root@pe-master modules (master)]# git log master..newfeature 
commit e8e6c5a5481b2e211a71f912a7185990bcdacf10
Author: root <root@pe-master.stiv.local>
Date:   Sun Jan 10 18:21:28 2016 +0100
    Added newfeature module

[root@pe-master modules (master)]# git merge newfeature 
Updating 755df81..e8e6c5a
Fast-forward
 modules/newfeature/manifests/init.pp | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 modules/newfeature/manifests/init.pp
[root@pe-master modules (master)]# git branch -d newfeature 
Deleted branch newfeature (was e8e6c5a).

Next step is the Merging of this "test master branch" to the "dev master branch".

[root@pe-master modules (master)]# cd ../../dev/
[root@pe-master dev (master)]# ls
environment.conf  hieradata  manifests  modules
[root@pe-master dev (master)]# git branch -r
  origin/HEAD -> origin/master
  origin/master
  test/master
[root@pe-master dev (master)]# git log master..test/master 
[root@pe-master dev (master)]# git fetch test 
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From /etc/puppetlabs/code/environments/test
   755df81..e8e6c5a  master     -> test/master
[root@pe-master dev (master)]# git log master..test/master 
commit e8e6c5a5481b2e211a71f912a7185990bcdacf10
Author: root <root@pe-master.stiv.local>
Date:   Sun Jan 10 18:21:28 2016 +0100
    Added newfeature module

[root@pe-master dev (master)]# git branch 
* master
[root@pe-master dev (master)]# git merge test/master 
Updating 755df81..e8e6c5a
Fast-forward
 modules/newfeature/manifests/init.pp | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 modules/newfeature/manifests/init.pp
[root@pe-master dev (master)]# git log master..test/master

We can now test that feature on Development System, and later on merge the dev and production repository to have the change live on the production systems.

[root@pe-dev ~]# puppet agent -t
Warning: Local environment: "production" doesn't match server specified node environment "dev", switching agent to "dev".
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for pe-dev.stiv.local
Info: Applying configuration version '1452446957'
Notice: This is the environment I am assigned to : dev
Notice: /Stage[main]/newfeature/Notify[This is the environment I am assigned to : dev]/message: defined 'message' as 'This is the environment I am assigned to : dev'
Notice: Applied catalog in 0.65 seconds

Just to summarize this section, once both production,dev and test Directory Environments repo are initially synchronized, the workflow (with git commands associated) is implemented as followed for any module modification/addition:

### From the test repository, Create New Branch for new feature/module
git checkout -b newmodule

### Develop Module in the normal way

### (with any number of commit in that newmodule branch) 
### and once satisfied with the new module/feature, still in the test repository, 
### merge our newmodule with our master branch            
git checkout master
git merge newmodule
                              
### Get to the dev repo, fetch the test repo in order to refresh

### our remote branch listing
git fetch test   

### Check what the remote branch has that isn't in the current master branch
git log master..test/master

### Merge the remote test branch with the production master branch
git merge test/master

### once satisfied with that Module on the Development Systems,
### we can move forward to our most critical systems (production), and merge dev
git fetch test
git fetch dev
git merge dev/master

### Get to the dev branch, fetch the origin repo in order to refresh its branch listing
git fetch origin

### Get to the test branch, fetch the origin
### and dev repo in order to refresh their branch listing
git fetch origin
git fetch dev

I.d: Some advantages and disadvantages of that Workflow:

All that seems good, and we can easily see what might be the advantages of such design; No changes pushed to production Systems without extensive tests, first in the test environment, then in the dev environment, Well written history of the evolution of our main environment... But they are also some (or many...) disadvantages using git repo and Puppet in this way. In fact, one of the main purpose of using Version Control Software (and git) is collaboration. But this configuration doesn't really help achieve that noble goal. To put that in simple ways,

1. How do we enable the tons of external awesome developers in the world to help enhance the module we're writing (in my case, how can I push some of my Puppet Module Code to Public github/gitlab account...)? We obviously can't push our whole Test repo to github and expects people to collaborate on that, we need a way to start a new (single) project/module, and push only that module to Public CVS.

2. How are we pulling Public Written Module which aren't on Forge (e.g: github,gitlab...) to our environment? If we do that, we'll be pulling another repository inside our repo (nested git repo) and that's a bit tough to deal with.

3. Eight (08) manual steps to have a Module modification on Production System! That seems a lot, and it's probably prone to mistake!

II.Moving to one repo per Puppet Module and a Centralized Git:

*** The Fellowship of the Ring

Having now these huge Test, dev and production repos which contain all our downloaded and written modules , I'm aiming to describe in this section how I'm moving from that to smaller repo which are more manageable and easily shareable. As described above, I have 03 similar environments; test,dev and production. Test is where I'm making most of the modification and dev and production are replicas of that repository (Mainly used for node classification.). For the split,I'm mainly working on that Test environment.

But before moving forward with the Test environment's split, there's a set of requirements that must be met:

The first requirement is to have another System that will be used as Centralized git System. It is possible to use a Public Central Repo for this, but In this case, there are some modules that should be kept Private (the one that are specific to the running environment) and some that will be shared publicly, so I found that having an Internal central Git System will be nice. As you will see below, I'm making use of GitLab for that Internal Git System.
The second is to make sure that we have no other branch than the master one, so we've to merge all our modification to the master branch of our Test repo.
The last requirement is to have a Backup of all our environments :-) . For that a simple tar will do the trick, something like :

  cd /etc/puppetlabs/code && tar zcvf /tmp/environments_before_split.tar.gz environments

II.a: Split the monolithic repo:

Once the little requirements described above are met, we can apply the recommended strategy for such split which is to make a full clone of the main repo (test) to another local directory named with each module's name. Then, for each of this module, use git filter-branch --subdirectory-filter command to select commits that apply only to that module and we're done.

If that seems a bit tough to understand, then let's see how it is done practically. First we need have a look on the number of modules we've to migrate by using the tree command as seen below.

[root@pe-master test (master)]# tree -d -L 2 modules
modules
├── apache
│   ├── files
│   ├── lib
│   ├── manifests
│   ├── spec
│   ├── templates
│   └── tests
├── banner
│   ├── manifests
│   ├── templates
│   └── tests
├── chrony
│   ├── manifests
│   ├── spec
│   ├── templates
│   └── tests
├── newfeature
│   ├── files
│   ├── manifests
│   └── tests
├── ntp
│   ├── lib
│   ├── manifests
│   ├── spec
│   ├── templates
│   └── tests
├── profiles
│   ├── files
│   ├── manifests
│   ├── templates
│   └── tests
├── roles
│   ├── files
│   ├── manifests
│   ├── templates
│   └── tests
├── stdlib
│   ├── examples
│   ├── lib
│   ├── manifests
│   └── spec

So,we have a bunch of Puppet Modules Installed from Puppet Forge (puppetlabs-apache, puppetlabs-ntp, puppetlabs-stdlib and ringingliberty-chrony ), we also have the roles and profiles module and two other Internal written Modules (banner and newfeature).

Let us first try to split a single module (newfeature) by following the described strategy which is (after that we will automate that process):

Create a new (temporary) working directory and get in that directory

Clone the test directory environment to a repo named after the module we are willing to split (

and remove origin after the cloning)

Keep only the history relevant to that module

Check that our module only contains the relevant subdirectory

###1.### Create the new working directory and get in that directory
[root@pe-master tmp]# cd /tmp/
[root@pe-master tmp]# mkdir modules
[root@pe-master tmp]# cd modules/

###2.### Clone the test directory environment, better remove origin after the cloning
[root@pe-master modules]# git clone /etc/puppetlabs/code/environments/test newfeature
Cloning into 'newfeature'...
done.
[root@pe-master modules]# cd newfeature/
[root@pe-master newfeature (master)]# git remote -v
origin  /etc/puppetlabs/code/environments/test (fetch)
origin  /etc/puppetlabs/code/environments/test (push)
[root@pe-master newfeature (master)]# git remote remove origin

###3.### Keep only the history relevant to that module
[root@pe-master newfeature (master)]# git filter-branch --subdirectory-filter modules/newfeature
Rewrite e8e6c5a5481b2e211a71f912a7185990bcdacf10 (1/1)
Ref 'refs/heads/master' was rewritten

###4.### Check that our module only contains the relevant subdirectory
[root@pe-master newfeature (master)]# cat manifests/init.pp 
class newfeature {
        notify { "This is the environment I am assigned to : ${::environment}": }
}
[root@pe-master newfeature (master)]# git status 
# On branch master
nothing to commit, working directory clean
[root@pe-master newfeature (master)]# git log --oneline
81ca0ca Added newfeature module

Having many modules to split using that same sequence, I think it's better to write a script for that activity. Here is a Python script that is written for that purpose and below output of that script in this environment (My strong suggestion for running such a script is to switch to an unprivileged system user who has only read right to the Directory Environment folder).

[stiv@pe-master ~]$ ./split_monolithic_puppet_git_repository.py /tmp/modules /etc/puppetlabs/code/environments/test
### Start Split for apache
running git clone /etc/puppetlabs/code/environments/test apache ...
Cloning into 'apache'...
done.