devsnack/control-repo
1
0
Fork 0
Code Releases Activity

Merge pull request #27 from npwalker/reduce_scope_of_control_repo

Browse Source 
Reduce the scope of the control-repo to mostly an example
This commit is contained in:
Gary Larizza 2016-03-29 16:36:03 -07:00
commit 3f94dafcd9
8 changed files with 54 additions and 290 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

61
Puppetfile
View File

@ -1,54 +1,17 @@
forge "http://forge.puppetlabs.com" forge "http://forge.puppetlabs.com"
# Modules from the Puppet Forge # Modules from the Puppet Forge
# Note the versions are all set to :latest but after you've # Versions should be updated to be the latest at the time you start
# installed you should change them to the exact version you want #mod "puppetlabs/inifile", '1.5.0'
mod "hunner/hiera", '2.0.1' #mod "puppetlabs/stdlib", '4.11.0'
mod "puppetlabs/inifile", :latest #mod "puppetlabs/concat", '2.1.0'
mod "puppetlabs/stdlib", :latest
mod "puppetlabs/concat", :latest
mod "puppetlabs/ntp", :latest
mod "saz/ssh", :latest
mod "puppetlabs/postgresql", '4.5.0'
mod "puppet/stash", '1.3.0'
mod "puppetlabs/java", '1.4.1'
mod "npwalker/pe_code_manager_webhook", '1.0.3'
#An example of using a specific forge module version instead of latest # Modules from Git
#Notice the addition of single quotes # Examples: https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd#examples
#mod "puppetlabs/inifile", '1.3.0' #mod 'apache',
# :git => 'https://github.com/puppetlabs/puppetlabs-apache',
# :commit => '83401079053dca11d61945bd9beef9ecf7576cbf'
# Modules from Github using various references #mod 'apache',
# Further examples: https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd#examples # :git => 'https://github.com/puppetlabs/puppetlabs-apache',
# update the tag to the most current release when implementing # :branch => 'docs_experiment'
mod 'gitlab',
:git => 'https://github.com/vshn/puppet-gitlab',
:ref => '00397b86dfb3487d9df768cbd3698d362132b5bf'
mod 'r10k',
:git => 'https://github.com/acidprime/r10k',
:tag => 'v3.1.1'
#mod 'pe_code_manager_webhook',
# :git => 'https://github.com/npwalker/pe_code_manager_webhook',
# :ref => '6957aa67e376ca7dcc9a60a58c247ed3e7b66c16'
mod 'gms',
:git => 'https://github.com/npwalker/abrader-gms',
:branch => 'gitlab_disable_ssl_verify_support'
mod 'pltraining-rbac',
:git => 'https://github.com/puppetlabs/pltraining-rbac',
:ref => '2f60e1789a721ce83f8df061e13f8bf81cd4e4ce'
mod 'rampup_profile_puppetmaster',
:git => 'https://github.com/PuppetLabs-RampUpProgram/RampUp_profile_puppetmaster',
:branch => 'master'
mod 'rampup_profile_gitlab',
:git => 'https://github.com/PuppetLabs-RampUpProgram/RampUp_profile_gitlab',
:branch => 'master'
mod 'rampup_profile_stash',
:git => 'https://github.com/PuppetLabs-RampUpProgram/RampUp_profile_stash',
:branch => 'master'

203
README.md
View File

@ -1,67 +1,45 @@
Table of Contents Table of Contents
================= =================
* [Before Starting](#before-starting) * [Where Did All The Previous Code Go?](#where-did-all-the-previous-code-go)
* [What You Get From This control\-repo](#what-you-get-from-this-control-repo) * [What You Get From This control\-repo](#what-you-get-from-this-control-repo)
* [How To Set It All Up](#how-to-set-it-all-up)
* [Setup a Trusted Fact On Your PE Master](#setup-a-trusted-fact-on-your-pe-master)
* [If You Have Not Installed PE](#if-you-have-not-installed-pe)
* [If You Have Already Installed PE](#if-you-have-already-installed-pe)
* [Copy This Repo Into Your Own Git Server](#copy-this-repo-into-your-own-git-server) * [Copy This Repo Into Your Own Git Server](#copy-this-repo-into-your-own-git-server)
* [Gitlab](#gitlab) * [Gitlab](#gitlab)
* [Stash](#stash) * [Stash](#stash)
* [Github](#github) * [Github](#github)
* [Configure PE to Use the Control\-Repo](#configure-pe-to-use-the-control-repo)
* [Install PE](#install-pe)
* [Get the Control\-Repo Deployed On Your Master](#get-the-control-repo-deployed-on-your-master)
* [Test Code Manager](#test-code-manager)
* [Updating From a Previous Version of PE](#updating-from-a-previous-version-of-pe)
* [Upgrading to PE2015\.3\.z from PE 2015\.2\.z](#upgrading-to-pe20153z-from-pe-20152z)
* [Appendix](#appendix)
* [Test The Zack/r10k Webhook](#test-the-zackr10k-webhook)
# Before Starting Created by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc.go)
This control repo and the steps below are intended to be used during a new installation of PE. # Where Did All The Previous Code Go?
The instructions are geared towards a new installation of PE2015.3.z. However, the control-repo should work just fine on [PE2015.2.z](#upgrading-to-pe20153z-from-pe-20152z) Initially, the control-repo project began as a 'starter' template for anyone
who wanted to get started with R10k. As time passed (and Code Manager was
If you intend to use this control-repo on an existing installation then be warned that if you've already written or downloaded modules when you start using r10k it will remove all of the existing modules and replace them with what you define in your Puppetfile. Please copy or move your existing modules to another directory to ensure you do not lose any work you've already started. integrated into Puppet Enterprise), the scope of this project grew to include
opinionated Puppet profiles to setup many Puppet Enterprise components. As the
code increased, so did the complexity of the control-repo project. To reduce
that complexity, as well as continue to meet the needs of individuals who would
like a more minimal template, this repository was stripped of anything other
than the bare minimum files necessary to get started with a functioning
control-repo. All of the code that was previously in this repository still
exists in separate repositories under the [Puppet Labs RampUp Program namespace within Github](https://github.com/PuppetLabs-RampUpProgram)
and can easily be re-connected to an existing control-repo if that is required
(simply add the modules to the Puppetfile). Alternatively, if that
previously-opinoinated control-repo is desired, [it still exists on Github under the Puppet Labs RampUp Program namespace.](https://github.com/PuppetLabs-RampUpProgram/control-repo)
This control-repo project will remain a template for anyone who would like a minimal
'starter' template.
# What You Get From This control-repo # What You Get From This control-repo
As a result of following the instructions below you will receive at least the beginning of a best-practices installation of PE including... This repository exists as a template control-repo that can be used with R10k or Puppet Enterprise Code Manager.
- A git server The major points are:
- The ability to push code to your git server and have it automatically deployed to your PE Master - An environment.conf that correctly implements:
- A config_version script to output the commit of code that your agent just applied - A site directory for roles, profiles, and any custom modules for your organization
- Optimal tuning of PE settings for this configuration - A config_version script
- Working and example roles/profiles code - Provided config_version scripts to output the commit of code that your agent just applied
- Basic example of roles/profiles code
# How To Set It All Up - Example hieradata directory with pre-created common.yaml and nodes directory
- These match the default hierarchy that ships with PE
## Setup a Trusted Fact On Your PE Master
This control repository is setup to manage certain portions of your PE installation for you if you create a trusted fact called `pp_role`. In order to do so, lay down a file that looks exactly like the below in `/etc/puppetlabs/puppet/csr_attributes.yaml`
```
---
extension_requests:
#pp_role
1.3.6.1.4.1.34380.1.1.13: 'all_in_one_pe'
```
### If You Have Not Installed PE
Good then you can proceed forward and the trusted fact will be used when you get to the install step.
### If You Have Already Installed PE
Trusted facts are created at the time a CSR is generated. So, we need to regenerate the certificate on the master for the above trusted fact to be created.
Follow this document to regenerate the certificate on your master.
http://docs.puppetlabs.com/pe/latest/regenerate_certs_master.html
##Copy This Repo Into Your Own Git Server ##Copy This Repo Into Your Own Git Server
@ -81,45 +59,23 @@ http://docs.puppetlabs.com/pe/latest/regenerate_certs_master.html
5. Create a group called `puppet` ( this is case sensitive ) 5. Create a group called `puppet` ( this is case sensitive )
- http://doc.gitlab.com/ce/workflow/groups.html - http://doc.gitlab.com/ce/workflow/groups.html
6. Create a user called `r10k_api_user` and add them to the `puppet` group 6. Add your user to the `puppet` group as well
- From the landing page, select groups
- Choose the puppet group
- In the left hand pane, select memembers
- Add the `r10k_api_user` with `master` permissions
7. Add your user to the `puppet` group as well 7. Create a project called `control-repo` and set the Namespace to be the `puppet` group
8. Create a project called `control-repo` and set the Namespace to be the `puppet` group 8. Clone this control repository to your laptop/workstation
9. Logout of root and login as the `r10k_api_user`
- Go to profile settings -> account ( https://<your_gitlab_server>/profile/account )
- Copy the api token
10. Clone this control repository to your laptop/workstation
- `git clone <repository url>` - `git clone <repository url>`
- `cd control-repo` - `cd control-repo`
11. `git mv hieradata/nodes/example-puppet-master.yaml hieradata/nodes/<fqdn_of_your_puppet_master>.yaml` 9. Remove this repository as the origin remote
- Open `hieradata/nodes/<fqdn_of_your_puppet_master>.yaml` - `git remote remove origin`
- edit `gms_api_token` to be your api token
- edit `git_management_system` to be 'gitlab'
- edit the `gms_server_url`
12. `git add hieradata/nodes/<fqdn_of_your_puppet_master>.yaml` 10. Add your internal repository as the origin remote
13. `git commit -m "renaming example-puppet-master.yaml"`
14. Rename my repository as the upstream remote
- `git remote rename origin upstream`
15. Add your internal repository as the origin remote
- `git remote add origin <url of your gitlab repository>` - `git remote add origin <url of your gitlab repository>`
16. Push the production branch of the repository from your machine up to your git server 11. Push the production branch of the repository from your machine up to your git server
- `git push origin production` - `git push origin production`
17. `git branch --set-upstream-to origin/production`
###Stash ###Stash
Coming soon! Coming soon!
@ -127,92 +83,3 @@ Coming soon!
###Github ###Github
Coming soon! Coming soon!
##Configure PE to Use the Control-Repo
###Install PE
1. Download the latest version of the PE installer for your platform and copy it to your master
- https://puppetlabs.com/download-puppet-enterprise
2. Expand the tarball and `cd` into the directory
3. Run `puppet-enterprise-installer` to install
If you run into any issues or have more questions about the installer you can see our docs here:
http://docs.puppetlabs.com/pe/latest/install_basic.html
###Get the Control-Repo Deployed On Your Master
At this point you have our control-repo code deployed into your git server. However, we have one final challenge: getting that code onto your puppet master. In the end state the master will pull code from the git server via r10k, however, at this moment your puppet master doesn't have credentials to get code from the git server.
So, we'll set up a deploy key in the git server that will allow a ssh-key we make to deploy the code and configure everything else.
1. On your puppet master, make an ssh key for r10k to connect to gitlab
- `/usr/bin/ssh-keygen -t rsa -b 2048 -C 'code_manager' -f /etc/puppetlabs/puppetserver/code_manager.key -q -N ''`
- http://doc.gitlab.com/ce/ssh/README.html
- https://help.github.com/articles/generating-ssh-keys/
2. Create a deploy key on the `control-repo` project in Gitlab
- Paste in the public key from above
- `cat /etc/puppetlabs/puppetserver/code_manager.key.pub`
3. Login to the PE console
7. Navigate to the Classification page
- Click on the PE Master group
- Click the Classes tab
- Add the `puppet_enterprise::profile::master`
- Set the `r10k_remote` to the ssh url from the front page of your gitlab repo
- Set the `r10k_private_key` parameter to `/etc/puppetlabs/puppetserver/code_manager.key`
- Commit your changes
8. Run `puppet agent -t`
- Expect to see changes to `r10k.yaml`
9. Run `r10k deploy environment -pv`
10. Run `puppet agent -t`
- Expect to see code manager enabled
10. `echo 'code_manager_mv_old_code=true' > /opt/puppetlabs/facter/facts.d/code_manager_mv_old_code.txt`
11. Run `puppet agent -t`
- Now you should see many more changes
- Your code has been deployed with code manager now
## Test Code Manager
One of the components setup by this control-repo is that when you "push" code to your git server, the git server will inform the puppet master to deploy branch you just pushed.
1. In one terminal window, `tail -f /var/log/puppetlabs/puppetserver/puppetserver.log`
2. In a second terminal window
- Add a new file, `touch test_file`
- `git add test_file`
- `git commit -m "adding a test_file"`
- `git push origin production`
3. Allow the push to complete and then wait a few seconds for everything to sync over
- `ls -l /etc/puppetlabs/code/environments/production`
- Confirm test_file is present
4. In your first terminal window review the `puppetserver.log` to see the type of logging each sync will create
----
# Updating From a Previous Version of PE
## Upgrading to PE2015.3.z from PE 2015.2.z
Remove `pe_r10k` from the PE master group in the console and instead add the following two parameters to the `puppet_enterprise::profile::master` class under the PE master group.
- `r10k_remote` = the ssh url for your internal repo
- `r10k_private_key` = `/etc/puppetlabs/puppetserver/code_manager.key`
When upgrading the `puppet_enterprise::profile::master` class has the `file_sync_enabled` parameter set to `false`. This parameter should be removed so that code manager can configure file sync.
Finally, you’ll need to `echo 'code_manager_mv_old_code=true' > /opt/puppetlabs/facter/facts.d/code_manager_mv_old_code.txt` so that my puppet code will redeploy all of your code with code manager.
# Appendix
## Test The Zack/r10k Webhook
If you are using PE2015.2.z or if you've forced the use of the zack/r10k webhook then you'll want to test that it works.
One of the components setup by this control-repo is that when you "push" code to your git server, the git server will inform the puppet master to run `r10k deploy environment -p`.
1. Edit README.md
- Just add something to it
2. `git add README.md`
3. `git commit -m "edit README"`
4. `git push origin production`
5. Allow the push to complete and then give it few seconds to complete
- Open `/etc/puppetlabs/code/environments/production/README.md` and confirm your change is present

8
hieradata/common.yaml
View File

@ -3,11 +3,3 @@ message: "This node is using common data"
#Puppet Server Tuning #Puppet Server Tuning
puppet_enterprise::master::puppetserver::jruby_max_requests_per_instance: 0 puppet_enterprise::master::puppetserver::jruby_max_requests_per_instance: 0
#Enable code manager
puppet_enterprise::profile::master::code_manager_auto_configure: true
puppet_enterprise::master::code_manager::authenticate_webhook: false
#pe-console-services tuning
#https://docs.puppetlabs.com/pe/latest/console_config.html#tuning-the-classifier-synchronization-period
#disable classifier scheduled sync and rely on r10k postrun command to sync the classes
puppet_enterprise::profile::console::classifier_synchronization_period: 0

1
hieradata/nodes/example-node.yaml Normal file
View File

@ -0,0 +1 @@
---

8
hieradata/nodes/example-puppet-master.yaml
View File

@ -1,8 +0,0 @@
---
git_management_system: 'gitlab'
gms_server_url: 'https://gitlab-server'
gms_api_token: 'BDkZfWWnk4LVTLHdAywd'
#setup r10k to update classes in the console after code deploy
pe_r10k::postrun:
- '/usr/local/bin/update-classes.sh'

42
hieradata/virtual/virtualbox.yaml
View File

@ -1,42 +0,0 @@
#These setting are intended for low memory testing VMs
#Not intended for general usage
---
#Drop puppetdb Java Heap Size
#PE3.2 and above
pe_puppetdb::pe::java_args:
-Xmx: '256m'
-Xms: '64m'
#PE3.1 and below
pe_puppetdb::java_args:
-Xmx: '256m'
-Xms: '64m'
#Drop the activemq java heap size
pe_mcollective::role::master::activemq_heap_mb: '96'
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
pe_puppetdb::pe::listen_address: '0.0.0.0'
#PE3.7+
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
puppet_enterprise::profile::puppetdb::listen_address: '0.0.0.0'
puppet_enterprise::profile::amq::broker::heap_mb: '96'
puppet_enterprise::profile::master::java_args:
Xmx: '192m'
Xms: '128m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::puppetdb::java_args:
Xmx: '96m'
Xms: '96m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::console::java_args:
Xmx: '64m'
Xms: '64m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1 #PE3.7.2 only
puppet_enterprise::profile::console::delayed_job_workers: 1
#shared_buffers takes affect during install but is not managed after
puppet_enterprise::profile::database::shared_buffers: '4MB'

8
manifests/site.pp
View File

@ -26,14 +26,6 @@ File { backup => false }
# specified in the console for that node. # specified in the console for that node.
node default { node default {
#incude a role on any node that specifies it's role via a trusted fact at provision time
#https://docs.puppetlabs.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#trusted-facts
#https://docs.puppetlabs.com/puppet/latest/reference/ssl_attributes_extensions.html#aws-attributes-and-extensions-population-example
if !empty( $trusted['extensions']['pp_role'] ) {
include "role::${trusted['extensions']['pp_role']}"
}
# This is where you can declare classes for all nodes. # This is where you can declare classes for all nodes.
# Example: # Example:
# class { 'my_class': } # class { 'my_class': }

1
site/role/manifests/all_in_one_pe.pp
View File

@ -1,6 +1,5 @@
class role::all_in_one_pe { class role::all_in_one_pe {
include rampup_profile_puppetmaster
include pe_code_manager_webhook include pe_code_manager_webhook
} }