I spent near month to figured out why i can’t update puppet on ubuntu 16 with specially designed puppet_agent module. It was task full of confusing experience.
So, let’s start. For a beginning you shouldn’t debug update process from a console, because one of a bug related to puppet service. You could solve all problem which you will found with ‘puppet agent -t’ but when you will try to upgrade puppet when it daemonized, it will fail. So set ‘log_level=info’ in your puppet.conf and use kill to trigger puppet daemon.
sudo kill -SIGUSR1 $(cat /var/run/puppet/agent.pid); |
Next you should set ‘stringify_facts=false’ into puppet.conf. Now puppet_agent developers declared that they provide additional class ‘::puppet_agent::prepare::stringify_facts’ for that, but when i started upgrade procedure it wasn’t available (or i miss it), so here is external fact to provide stringify_facts settings and puppet.conf path:
require 'puppet' Facter.add('puppet_config') do setcode do Puppet.settings['config'] end end Facter.add('puppet_stringify_facts') do setcode do Puppet.settings['stringify_facts'] || false end end |
Call it something like puppet.rb and put it into <YOURMODULEDIR>/lib/facter.
Next puppet code will disable stringify_facts before doing upgrade:
if versioncmp($::clientversion, '4') < 0 { if $::puppet_stringify_facts { augeas { 'puppet.conf.stringify_facts': context => "/files${::puppet_config}/main", changes => [ 'set stringify_facts false', ], } } else { <Do puppet upgrade here> } |
If you have puppet service defined somewhere, you will be faced with duplicate service declaration:
Feb 16 09:09:59 localhost puppet-agent[10026]: Could not retrieve catalog from remote server: Error 500 on SERVER: {"message":"Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Service[puppet] is already declared in file CUT:47; cannot redeclare at /etc/puppetlabs/code/environments/production_puppet4/modules/puppet_agent/manifests/service.pp:31 at /etc/puppetlabs/code/environments/production_puppet4/modules/puppet_agent/manifests/service.pp:31:7 on node llocalhost","issue_kind":"RUNTIME_ERROR","stacktrace":["Warning: The 'stacktrace' property is deprecated and will be removed in a future version of Puppet. For security reasons, stacktraces are not returned with Puppet HTTP Error responses."]} |
So you should declare puppet_agent class in next maner:
class {'::puppet_agent': collection => 'PC1', service_names => [], notify => Service['puppet'] } |
Interesting what will happens now if you will try to update puppet?
Feb 16 09:16:32 localhost puppet-agent[10474]: Caught TERM; exiting Feb 16 09:16:32 localhost puppet-agent[8171]: Caught TERM; exiting Feb 16 09:16:32 localhost systemd[1]: Stopping Puppet agent... Feb 16 09:16:36 localhost systemd[1]: Stopped Puppet agent. |
Tadaaam. Now you have barely installed puppet-agent package, deleted previous puppet package and killed puppet daemon:
ichurkin@localhost:~$ pgrep -f puppet ichurkin@localhost:~$ dpkg -l|grep puppet rF puppet 3.8.5-2 all configuration management system, agent ii puppet-common 3.8.5-2 all configuration management system |
It happens because during puppet-agent package installation systemd killed puppet daemon and all its children. So you need to fix unit file first:
[Service] KillMode=process |
Call it something like service.override.conf and put into <YOURMODULEDIR>/files, puppet code to fix that:
if $::os['name'] == 'Ubuntu' and versioncmp($::os['release']['major'], '16') >= 0 { notify{ "Creating systemd ovveride file":} file {'/etc/systemd/system/puppet.service.d/': ensure => directory }~> file { '/etc/systemd/system/puppet.service.d/override.conf': mode => '0644', owner => 'root', group => 'root', source => 'puppet:///modules/puppet/puppet.service.override', }~> exec { 'systemd_reload': command => 'systemctl daemon-reload', path => [ '/usr/bin', '/bin', '/sbin', '/usr/sbin' ], refreshonly => true, before => Class['::puppet_agent'] } |
I tried to use fact ${::service_provider} instead of ugly os/release condition, but at least puppet 3.8 on ubuntu 16 return ‘debian’ instead of ‘systemd’.
Let’s update puppet?
Feb 16 04:49:14 localhost puppet-agent[10021]: Could not start Service[puppet]: Execution of '/usr/sbin/service puppet start' returned 1: Failed to start puppet.service: Unit puppet.service is masked. Feb 16 04:49:14 localhost puppet-agent[10021]: (/Stage[main]/Puppet_agent::Service/Service[puppet]/ensure) change from stopped to running failed: Could not start Service[puppet]: Execution of '/usr/sbin/service puppet start' returned 1: Failed to start puppet.service: Unit puppet.service is masked. |
Once again puppet render itself stopped, i think it may caused because service provider is debian instead of systemd, i too exhausted to search for right solution, so here another one dirty hack:
exec { 'puppetagent_transition_restart': path => '/bin:/sbin:/usr/bin:/usr/sbin', command => '/opt/puppetlabs/bin/puppet resource service puppet enable=true ensure=running', require => Class['::puppet_agent'] } |
That’s all.
PS
List of related bugs below:
https://tickets.puppetlabs.com/browse/MODULES-3453
https://tickets.puppetlabs.com/browse/PUP-5637
https://tickets.puppetlabs.com/browse/PUP-3931
https://github.com/puppetlabs/puppet/pull/3699
https://github.com/puppetlabs/puppet/pull/3700
https://tickets.puppetlabs.com/browse/PUP-4512