Merge branch 'production' of https://github.com/npwalker/control-repo into merge_npwalker_control_repo

This commit is contained in:
Nick Walker 2015-11-24 11:16:51 -08:00
commit afc9a358f3
20 changed files with 469 additions and 20 deletions

1
.gitignore vendored Normal file
View File

@ -0,0 +1 @@
hieradata/nodes/example-puppet-master.yaml

View File

@ -6,6 +6,11 @@ forge "http://forge.puppetlabs.com"
mod "puppetlabs/inifile", :latest
mod "puppetlabs/stdlib", :latest
mod "puppetlabs/concat", :latest
mod "puppetlabs/ntp", :latest
mod "saz/ssh", :latest
mod "puppetlabs/postgresql", '4.5.0'
mod "puppet/stash", '1.3.0'
mod "puppetlabs/java", '1.4.1'
#An example of using a specific forge module version instead of latest
#Notice the addition of single quotes
@ -14,15 +19,18 @@ mod "puppetlabs/concat", :latest
# Modules from Github using various references
# Further examples: https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd#examples
# update the tag to the most current release when implementing
#mod 'hiera',
# :git => 'https://github.com/hunner/puppet-hiera',
# :tag => '1.3.1'
mod 'hiera',
:git => 'https://github.com/hunner/puppet-hiera',
:tag => '1.3.1'
#mod 'notifyme',
# :git => 'git://github.com/glarizza/puppet-notifyme',
# :ref => '50c01703b2e3e352520a9a2271ea4947fe17a51f'
mod 'gitlab',
:git => 'https://github.com/vshn/puppet-gitlab',
:ref => '00397b86dfb3487d9df768cbd3698d362132b5bf'
#mod 'profiles',
# :git => 'git://github.com/glarizza/puppet-profiles',
# :ref => '3611ae4253ff01762f9bda1d93620edf8f9a3b22'
mod 'r10k',
:git => 'https://github.com/acidprime/r10k',
:tag => 'v3.1.1'
mod 'gms',
:git => 'https://github.com/npwalker/abrader-gms',
:branch => 'gitlab_disable_ssl_verify_support'

177
README.md
View File

@ -1,2 +1,179 @@
# control-repo
A control repository template
=======
# Before Starting:
This control repo and the steps below are intended to be used during a new installation of PE.
If you intend to use it on an existing installation then be warned that if you've already written or downloaded modules when you start using r10k it will remove all of the existing modules and replace them with what you define in your Puppetfile. Please copy or move your existing modules to another directory to ensure you do not lose any work you've already started.
## Setup a Trusted Fact On Your PE Master
This control repository is setup to manage certain portions of your PE installation for you if you create a trusted fact called `pp_role`. In order to do so, lay down a file that looks exactly like the below in `/etc/puppetlabs/puppet/csr_attributes.yaml`
```
---
extension_requests:
#pp_role
1.3.6.1.4.1.34380.1.1.13: 'all_in_one_pe'
```
### If You Have Not Installed PE
Good then you can proceed forward and the trusted fact will be used when you get to the install step.
### If You Have Already Installed PE
Trusted facts are created at the time a CSR is generated. So, we need to regenerate the certificate on the master for the above trusted fact to be created.
Follow this document to regenerate the certificate on your master.
http://docs.puppetlabs.com/pe/latest/regenerate_certs_master.html
##Copy This Repo Into Your Own Git Server
###Gitlab
1. Install Gitlab
- https://about.gitlab.com/downloads/
2. After Gitlab is installed you may sign if with the `root` user and password `5iveL!fe`
3. Make an user for yourself
4. Make an ssh key to link with your user. Youll want to do this on the machine you intend to edit code from ( most likely not your puppet master but your local workstation / laptop )
- http://doc.gitlab.com/ce/ssh/README.html
- https://help.github.com/articles/generating-ssh-keys/
5. Create a group called `puppet` ( this is case sensitive )
- http://doc.gitlab.com/ce/workflow/groups.html
6. Create a user called `r10k_api_user` and add them to the `puppet` group
- From the landing page, select groups
- Choose the puppet group
- In the left hand pane, select memembers
- Add the `r10k_api_user` with `master` permissions
7. Add your user to the `puppet` group as well
8. Create a project called `control-repo` and set the Namespace to be the `puppet` group
9. Logout of root and login as the `r10k_api_user`
- Go to profile settings -> account ( https://<your_gitlab_server>/profile/account )
- Copy the api token
10. Clone this control repository to your laptop/workstation
- `git clone https://github.com/npwalker/control-repo.git`
- `cd control-repo`
11. `git mv hieradata/nodes/example-puppet-master.yaml hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`
- Open `hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`
- edit `gms_api_token` to be your api token
- edit `git_management_system` to be 'gitlab'
- edit the `gms_server_url`
12. `git add hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`
13. `git commit -m "renaming example-puppet-master.yaml"`
14. Rename my repository as the upstream remote
- `git remote rename origin upstream`
15. Add your internal repository as the origin remote
- `git remote add origin <url of your gitlab repository>`
16. Push the production branch of the repository from your machine up to your git server
- `git push origin production`
17. `git branch --set-upstream-to origin/production`
###Stash
Coming soon!
###Github
Coming soon!
##Configure PE to Use the Control-Repo
###Install PE
1. Download the latest version of the PE installer for your platform and copy it to your master
- https://puppetlabs.com/download-puppet-enterprise
2. Expand the tarball and `cd` into the directory
3. Run `puppet-enterprise-installer` to install
If you run into any issues or have more questions about the installer you can see our docs here:
http://docs.puppetlabs.com/pe/latest/install_basic.html
###Get the Control-Repo Deployed On Your Master
At this point you have my control-repo code deployed into your git server. However, we have one final challenge getting that code onto your puppet master. In the end state the master will pull code from the git server via r10k, however, at this moment your puppet master doesn't have credentials to get code from the git server.
So, we'll set up a deploy key in the git server that will allow a ssh-key we make to deploy the code and configure everything else.
1. On your puppet master, make an ssh key for r10k to connect to gitlab
- `/usr/bin/ssh-keygen -t rsa -b 2048 -C 'r10k' -f /root/.ssh/r10k_rsa -q -N ''`
- http://doc.gitlab.com/ce/ssh/README.html
- https://help.github.com/articles/generating-ssh-keys/
2. Create a deploy key on the `control-repo` project in Gitlab
- Paste in the public key from above
- `cat /root/.ssh/r10k_rsa.pub`
3. Follow https://docs.puppetlabs.com/pe/latest/r10k_config_console.html
- The remote is on the front page of the project in the gitlab UI
- git_settings should be:
- `{"provider": "rugged",
"private_key": "/root/.ssh/r10k_rsa"}`
3. Run `puppet agent -t`
- Expect to see changes to `r10k.yaml`
3. Run `r10k deploy environment -pv`
4. Run `puppet agent -t`
- Now you should see many more changes
## Test The Zack/r10k Webhook
One of the components setup by this control-repo is that when you "push" code to your git server, the git server will inform the puppet master to run `r10k deploy environment -p`.
1. Edit README.md
- Just add something to it
2. `git add README.md`
3. `git commit -m "edit README"`
4. `git push origin production`
5. Allow the push to complete and then give it few seconds to complete
- Open `/etc/puppetlabs/code/environments/production/README.md` and confirm your change is present
----
#Miscellaneous
## If You Want to Install Pointing To This Repo on Github
### Setting Up Gitlab
1. Install Gitlab on a server by specifying the following trusted fact on the soon-to-be Gitlab server and then [install the PE agent](http://docs.puppetlabs.com/pe/latest/install_agents.html#using-the-puppet-agent-package-installation-script).
```
---
extension_requests:
#pp_role
1.3.6.1.4.1.34380.1.1.13: 'gitlab'
```
### Setting up Github
Not yet completed.
### Setting up Stash
Not yet completed.
#TODO
Flush out generating an answer file and then appending extra answers onto the end of it.

View File

@ -1,8 +1,2 @@
modulepath = modules:site:$basemodulepath
config_version = 'scripts/config_version.sh $environmentpath $environment'
# Environment timeout should be set to unlimited. When set to zero it is less performant.
# When code is deployed the admin API of puppetserver should be used to force a refresh of code from disk.
# https://docs.puppetlabs.com/puppetserver/1.0/release_notes.html#new-feature-admin-api-for-refreshing-environments
# https://docs.puppetlabs.com/puppet/3.7/reference/environments_configuring.html#environmenttimeout
environment_timeout = 0

View File

@ -1,2 +1,10 @@
---
message: "This node is using common data"
#Puppet Server Tuning
puppet_enterprise::master::puppetserver::jruby_max_requests_per_instance: 10000
#pe-console-services tuning
#https://docs.puppetlabs.com/pe/latest/console_config.html#tuning-the-classifier-synchronization-period
#disable classifier scheduled sync and rely on r10k postrun command to sync the classes
puppet_enterprise::profile::console::classifier_synchronization_period: 0

View File

@ -0,0 +1,8 @@
---
git_management_system: 'gitlab'
gms_server_url: 'https://gitlab-server'
gms_api_token: 'BDkZfWWnk4LVTLHdAywd'
#setup r10k to update classes in the console after code deploy
pe_r10k::postrun:
- '/usr/local/bin/update-classes.sh'

View File

@ -0,0 +1,42 @@
#These setting are intended for low memory testing VMs
#Not intended for general usage
---
#Drop puppetdb Java Heap Size
#PE3.2 and above
pe_puppetdb::pe::java_args:
-Xmx: '256m'
-Xms: '64m'
#PE3.1 and below
pe_puppetdb::java_args:
-Xmx: '256m'
-Xms: '64m'
#Drop the activemq java heap size
pe_mcollective::role::master::activemq_heap_mb: '96'
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
pe_puppetdb::pe::listen_address: '0.0.0.0'
#PE3.7+
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
puppet_enterprise::profile::puppetdb::listen_address: '0.0.0.0'
puppet_enterprise::profile::amq::broker::heap_mb: '96'
puppet_enterprise::profile::master::java_args:
Xmx: '192m'
Xms: '128m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::puppetdb::java_args:
Xmx: '96m'
Xms: '96m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::console::java_args:
Xmx: '64m'
Xms: '64m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1 #PE3.7.2 only
puppet_enterprise::profile::console::delayed_job_workers: 1
#shared_buffers takes affect during install but is not managed after
puppet_enterprise::profile::database::shared_buffers: '4MB'

View File

@ -19,9 +19,7 @@
# Define filebucket 'main':
filebucket { 'main':
#server should point to one master that will be the file bucket
#$servername is the closest thing to the current behavior of PE
#which hardcodes each master to point to itself for a file buckket
server => $servername,
server => "${settings::server}",
path => false,
}
@ -39,6 +37,14 @@ File { backup => 'main' }
# specified in the console for that node.
node default {
#incude a role on any node that specifies it's role via a trusted fact at provision time
#https://docs.puppetlabs.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#trusted-facts
#https://docs.puppetlabs.com/puppet/latest/reference/ssl_attributes_extensions.html#aws-attributes-and-extensions-population-example
if !empty( $trusted['extensions']['pp_role'] ) {
include "role::${trusted['extensions']['pp_role']}"
}
# This is where you can declare classes for all nodes.
# Example:
# class { 'my_class': }

View File

@ -14,4 +14,4 @@ commit = repo.lookup(head_sha)
#add something to find the remote url
puts head_sha
puts head_sha

View File

@ -4,4 +4,4 @@ then
/opt/puppetlabs/puppet/bin/ruby $1/$2/scripts/config_version.rb $1 $2
else
/usr/bin/git --git-dir $1/$2/.git rev-parse HEAD
fi
fi

View File

@ -0,0 +1,11 @@
#https://docs.puppetlabs.com/pe/latest/nc_update_classes.html#post-v1update-classes
CONFDIR=$(puppet master --configprint confdir)
CERT=$(puppet master --confdir ${CONFDIR} --configprint hostcert)
CACERT=$(puppet master --confdir ${CONFDIR} --configprint localcacert)
PRVKEY=$(puppet master --confdir ${CONFDIR} --configprint hostprivkey)
OPTIONS="--cert ${CERT} --cacert ${CACERT} --key ${PRVKEY}"
CONSOLE=$(awk '/server: /{print $NF}' ${CONFDIR}/classifier.yaml)
curl -k -X POST ${OPTIONS} "https://${CONSOLE}:4433/classifier-api/v1/update-classes"

View File

@ -0,0 +1,5 @@
class profile::base {
#the base profile should include component modules that will be on all nodes
}

View File

@ -0,0 +1,24 @@
class profile::gitlab {
file { ['/etc/gitlab', '/etc/gitlab/ssl'] :
ensure => directory,
}
file { "/etc/gitlab/ssl/${::fqdn}.key" :
ensure => file,
source => "${::settings::privatekeydir}/${::trusted['certname']}.pem",
notify => Exec['gitlab_reconfigure'],
}
file { "/etc/gitlab/ssl/${::fqdn}.crt" :
ensure => file,
source => "${::settings::certdir}/${::trusted['certname']}.pem",
notify => Exec['gitlab_reconfigure'],
}
class { 'gitlab':
external_url => hiera( 'gms_server_url', "https://${::fqdn}") ,
require => File["/etc/gitlab/ssl/${::fqdn}.key", "/etc/gitlab/ssl/${::fqdn}.key"],
}
}

View File

@ -0,0 +1,73 @@
class profile::puppetmaster (
$webhook_username,
$webhook_password
) {
class { 'hiera':
hierarchy => [
'virtual/%{::virtual}',
'nodes/%{::trusted.certname}',
'common',
],
hiera_yaml => '/etc/puppetlabs/code/hiera.yaml',
datadir => '/etc/puppetlabs/code/environments/%{environment}/hieradata',
owner => 'pe-puppet',
group => 'pe-puppet',
notify => Service['pe-puppetserver'],
}
#BEGIN - Generate an SSH key for r10k to connect to git
$r10k_ssh_key_file = '/root/.ssh/r10k_rsa'
exec { 'create r10k ssh key' :
command => "/usr/bin/ssh-keygen -t rsa -b 2048 -C 'r10k' -f ${r10k_ssh_key_file} -q -N ''",
creates => $r10k_ssh_key_file,
}
#END - Generate an SSH key for r10k to connect to git
#BEGIN - Add deploy key and webook to git management system
$git_management_system = hiera('git_management_system', '')
if $git_management_system in ['gitlab', 'github'] {
git_deploy_key { "add_deploy_key_to_puppet_control-${::fqdn}":
ensure => present,
name => $::fqdn,
path => "${r10k_ssh_key_file}.pub",
token => hiera('gms_api_token'),
project_name => 'puppet/control-repo',
server_url => hiera('gms_server_url'),
provider => $git_management_system,
}
git_webhook { "web_post_receive_webhook-${::fqdn}" :
ensure => present,
webhook_url => "https://${webhook_username}:${webhook_password}@${::fqdn}:8088/payload",
token => hiera('gms_api_token'),
project_name => 'puppet/control-repo',
server_url => hiera('gms_server_url'),
provider => $git_management_system,
disable_ssl_verify => true,
}
}
#END - Add deploy key and webhook to git management system
#Lay down update-classes.sh for use in r10k postrun_command
#This is configured via the pe_r10k::postrun key in hiera
file { '/usr/local/bin/update-classes.sh' :
ensure => file,
source => 'puppet:///modules/profile/puppetmaster/update-classes.sh',
mode => '0755',
}
#https://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html#environmenttimeout
ini_setting { 'environment_timeout = unlimited':
ensure => present,
path => '/etc/puppetlabs/puppet/puppet.conf',
section => 'main',
setting => 'environment_timeout',
value => 'unlimited',
notify => Service['pe-puppetserver'],
}
}

View File

@ -0,0 +1,28 @@
class profile::stash {
class { 'java' :
version => present,
} ->
class { 'postgresql::globals':
manage_package_repo => true,
version => '9.4',
}->
class { 'postgresql::server': } ->
postgresql::server::db { 'stash':
user => 'stash',
password => postgresql_password('stash', 'password'),
} ->
class { 'stash':
javahome => '/etc/alternatives/java_sdk',
#dev.mode grants a 24-hour license for testing
java_opts => '-Datlassian.dev.mode=true',
}
file { '/opt/puppetlabs/bin/stash_mco.rb':
source => 'puppet:///modules/r10k/stash_mco.rb',
}
}

View File

@ -0,0 +1,29 @@
class profile::zack_r10k_webhook (
$username,
$password,
$use_mcollective = false,
) {
if $use_mcollective {
class { 'r10k::mcollective':
notify => Service['mcollective'],
}
}
class {'r10k::webhook::config':
enable_ssl => true,
protected => true,
user => $username,
pass => $password,
use_mcollective => $use_mcollective,
}
class {'r10k::webhook':
user => 'root',
group => '0',
require => Class['r10k::webhook::config'],
}
}

View File

@ -0,0 +1,16 @@
class role::all_in_one_pe {
$webhook_username = hiera('webhook_username', fqdn_rand_string(10, '', 'username'))
$webhook_password = hiera('webhook_password', fqdn_rand_string(20, '', 'password'))
class { 'profile::puppetmaster' :
webhook_username => $webhook_username,
webhook_password => $webhook_password,
}
class { 'profile::zack_r10k_webhook' :
username => $webhook_username,
password => $webhook_password,
}
}

View File

@ -0,0 +1,7 @@
class role::database_server {
#This role would be made of all the profiles that need to be included to make a database server work
#All roles should include the base profile
include profile::base
}

View File

@ -0,0 +1,5 @@
class role::gitlab {
include profile::gitlab
}

View File

@ -0,0 +1,7 @@
class role::webserver {
#This role would be made of all the profiles that need to be included to make a webserver work
#All roles should include the base profile
include profile::base
}