In this post we are going to learn a lot of things about puppet catalog, catalog compilation and how it is utilized by puppet master , It is a great know how for anyone interested in puppet and also a great resource for someone appearing for puppet dev/admin interview.

What is puppet catalog anyways ?

A catalog is a yaml document that describes the state of a puppet managed server at any given time, it contains all managed resources for that server, as well as any interdependencies between listed resources.

It looks like below, for complete catalog file content, please visit https://github.com/faintdream/misc/blob/master/node.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

--- !ruby/object:Puppet::Resource::Catalog
tags:
- settings
- node
- buildtools
- buildtools::wget
- wget
- buildtools::centos
- centos
- apache
- apache::download
- download
- apache::compile
- compile
- apache::service
- service
- apache::centos
- hierasample::lookup
- hierasample
- lookup
- hierasample::class1
- class1
- hierasample::class2
- class2
- hierasample::class3
- class3
- class

If you look closely you will figure that couple of classes are ready to be applied to the server like buildtools, apache::download hierasample & so on. but the question arises how did puppet master know that these resources are required to be applied to the server running puppet agent ? Simply put node itself informs a lot about what all it needs directly and indirectly through manifests.

Tip : to generate the catalog for your puppet managed node ( including puppet master) , simply run following command.

1
2

[root@node misc]# puppet catalog download
Notice: Saved catalog for node to /opt/puppetlabs/puppet/cache/client_yaml/catalog/node.yaml

For a puppet master to create node specific catalog, puppet master depends on following sources for truth,

1. Agent provided data
2. External Data
3. Manifest/Modules

Agent provided data

This is the set of information collected from the node running puppet agent.

• Node’s name same as node’s cert name and is embedded in request URI /puppet/v3/catalog/ubuntu.example.com?environment=production . if you remember when we run puppet agent -t for the first time, it creates as ssl certificate for itself and this certificate needs to be authorized by puppet master for both to be able to talk to each other ( https communication)
• Node certificate contains some additional information used for policy based autosigning & adding new trusted facts.
  [note: not applicable to masterless/stand alone architecture]
• Node’s facts including builtin as well as custom facts, some of these facts we use quite often in puppet code like $[os][family] , $[trusted][certname].
• Before requesting catalog , the agent requests its environment from the master, if master provides none, the environment is picked from agent’s config [/etc/pupetlabs/puppet.conf], this means the environment set on agent side is given least priority, so dont assume you will set something agent side and it will persist if there is already an environment set by the puppet master.

External data
Puppet uses two types of external data

• ENC - This is the data collected from Enhanced Node classifier or popularly known as ‘ENC’. ENC script can be carved out of your favourite programming language( python & ruby being most sought after in this use case) as long as it is designed to look for passed value and capture corresponding node information and returning the same to puppet master. The captured data is in the form of node object and may contain classes, node’s top scope variable, class configuration parameters/Environment information from master.
• Data from other sources, which can be invoked from main manifest or classes or defined types in modules. This kind of data includes,
  • Exported resources from PuppetDB.
  • The result of functions, which can access data sources including Hiera or an external configuration management database.

Manifest/Modules

manifests contain DSL (Declarative Style Language) that describes state of resource on target server.

catalog compilation process

A brief run down of what all happens during catalog compilation process,

1. node running puppet agent sends its certificate, facts & environment to puppet master.
2. puppet master requests node_terminus for a node object.
   • If the node_terminus is ‘plain’ , an empty node object is returned.
   • If the node_terminus is ‘exec’, a request is sent to ENC ( 3rd party script/software) to provide node specific data like what classes are to be applied etc.
   • If node_terminus is ‘ldap’. Node data is fetched from ldap db.
3. Set variables from node object, facts and certificate
   • This data is used by manifests in subsequent compilation stage.
   • Node facts are set as top scope variable.
   • The node’s facts are set in $facts hash( it remains immutable after this for the corresponding node) .
   • Some data from certificate header is set in the protected $trusted hash.
   • Any variable provided by the puppet master are set .
4. manifest evaluation
   1. Puppet parses the main manifest.
      • If there are node definition in the manifests, it must find a matching current node name for a catalog to compile, otherwise it fails compilation.
   2. Code outside the node definition is evaluated ( modules/classes/templates), resources in the code are added to the catalog and any classes declared in the code are loaded.
      • If a node is found in main manifest, code inside the node definition is evaluated at node level and any classes declared are loaded
   3. Evaluates classes from module .
      • If the classes were declared but not defined, puppet master looks at $modulepath to load the classes .
   4. Evaluate classes from node object
      • The process is the same as how classes are loaded from node definition or loading classes from modules, the only difference here is that the applicable classes were populated from node object.

That’s pretty much it , once the catalog is compiled it is thrown back to the node running agent in order to get the changes applied,

Introduction

I frequently write puppet modules so i always have to have a working puppet test environment with multiple nodes. while the instructions for setting up puppet server and agent are no way difficult but require some amount of manual work, which is not much fun and overtime virtual machines running on laptop go unstable due to no. of reasons, including me trying crazy things :-) . So everytime i start a new project it consumes an hour or so before i can even start being productive, this lead me to write Vagrantfile file to automate the process.

Using the Vagrantfile we can quickly spin up three VMs [ a puppet master & two nodes running puppet agent ] with a simple command, it still takes fewer minutes but far less compared to previous lead time.

Advantages include,

1. Fast vm provisioning
2. Repeatable process
3. Accuracy

Installation

Win/Mac/Linux ( any flavour)

Install following packages

1. Virtual box
2. Vagrant
3. Git

• Launch CMD/Terminal/Console on your system and run vagrant init, if this works we are good to proceed to next step.

• Let’s begin setting up puppet test environment,
  1. git clone https://github.com/faintdream/puppet55.git
  2. cd puppe55
  3. vagrant up

The new puppet test environment is up , but puppet agent running on the VMs are still not configured to talk to puppet master/puppetserver

• Establish HTTPS connectivity between all servers running puppet agent to server running puppet master/puppetserver

Commands to run on each server running puppet agent

1. vagrant ssh <vm name>

2. sudo su

3. puppet agent -t

   This will generate a client side ssl and will be used by puppetserver to authenticate the vm as a valid node.

   

   

Commands to run on server running puppet master/puppetserver

1. puppet cert sign --all

   this will sign all pending certificate requests, that means all nodes can talk to puppetserver

   

A working puppet test environment is now ready !

Bonus points if you get around this

To apply an exisitng apache module on node run puppet agent -t as root, this should install apache on the server.

1. vagrant ssh node
2. sudo su
3. puppet agent -t

Note: during bootstrap we downloaded several sample modules i created for demo purpose via r10k on puppetserver , if you are curious how the modules got deployed on puppetserver please look through Vagrantfile as well as Puppetfile under r10k-site.

Vagrant command reference:

Troubleshooting

• if you get below certificate error, it means that certificate is not working, we need to delete it as instructed in error output from the server running puppet agent as well as from the puppet master/puppetserver