Developer Automation Tools Bootcamp

Introduction to Chef

Of all the bootcamp subjects to cover this one is the most difficult to present properly in one short session. We will be able to cover "Why Chef exists," and "What Chef is," but not as much about "How Chef does it". That said, with the info here you will create a simple cookbook to configure a server.

What problems does Chef solve?

• How to specify what a system does as a description of state, instead of an aggregation of Bash/Perl/Python scripts.
• How to maintain many systems over time in a consistent, repeatable manner (where 'many' is anywhere between 1 and 250,000 or infinity)
• How to assure that machines in identical roles behave the same in different environments, e.g. between dev, staging and production.
• How to verify and test your changes before touching anything in your live infrastructure.

Chef is a product from Chef

• Chef will install and configure software for your server (or VM of a server) - or even your laptop if you wanted.
• Chef is a Ruby DSL (domain-specific language). Knowing Ruby is helpful but not required.
• Almost everything from Chef is open source. Including the community cookbooks Chef maintains.
• You write recipes which are stored in cookbooks. A knife does most of the work. (LOTS of cooking-related names.)
• It is assumed you are using version-control with Chef, generally Git. Git is not required, but some tools in the Chef ecosystem will actually try and execute Git commands.
• Chef and its ecosystem of tools are advancing at a pace I've rarely seen in a tool. It is *hard* to keep to up date.
• And you do not necessarily need to keep up to date if you have a functional environment or workflow.  But in general, try to.
• Wanna learn how to do something in a Chef cookbook? Cookbooks in the Chef Supermarket have examples of just about everything. Don't reinvent the wheel.

All the best practices of software development are available

The Ruby community is excellent at the practice of TDD (Test Driven Development). This has, thankfully, carried over to the Chef community, which has developed a suite of testing tools. These tools may add some time to your development initially, but their use pays off quickly, and they help increase both your velocity and stability in the long run.

• Dependency management - Berkshelf
• Code linting - Foodcritic and Cookstyle (born from Rubocop)
• Unit Testing - ChefSpec
• Integration Testing - by combining InSpec with Test Kitchen
• All these tools are so useful, Chef bundled them together into the Chef DK.
• Automation of linting, unit testing and interation testing after a code check in can be done in Countinuous Integration tools like Jenkins, just like building other software projects
• Successful completion of the above steps within your CI server would indicate a working cookbook/role/environment/data bag change and your CI server can then publish the change to the Chef server for managed machines to pull down the update and apply it.

Getting started

LearnChef has well-crafted tutorials based on your prefered client OS, as well as numerous other webinars and tutorials. The site provisions a virtual machine for you during the guided tours. Follow the steps as described and you'll accomplish more in an hour than you'd expect!

Chef Server/Client

All of the configuration of your node is done by the Chef Client. That client (sometimes called an agent) processes a runlist of cookbooks, evaluates them based on the node's attributes, and then fixes any resources on the node that aren't yet matching the cookbook description.

The Chef Server stores the cookbooks and node information for Chef Clients. The Chef Clients interact with the Chef Server over a REST API, and when you need to update cookbooks or other site information you use that API too, through tools like knife or the server's web Manage UI.

Whether you are using knife or the Chef Server's Manage UI, they will end up showing you the same information as gathered from the Server via those REST API requests. In some mature Chef environments users will not even have the Chef Manage UI installed on their Chef Server and instead they just use knife commands to request the information they need to know about their Chef managed nodes.

Do not reinvent the wheel

Now that we've covered that - here's the secret to Chef. It's all been done before!!! In the same way that Java has Maven, Ruby has gems or Node has npm for getting libraries people have already created - Chef has community cookbooks. They're open source - use them *and* learn from them.

Need Java? Use the community cookbook!
Need Tomcat? Use the community cookbook!
Need MongoDB? Use the community cookbook!

For a generic install of MongoDB using the community cookbook it is two lines of code added to your cookbook.

# in metadata.rb file
depends 'mongodb'

# in default.rb file
include_recipe 'mongodb'

DONE AND DONE. MongoDB is installed.

Do not reinvent the wheel (unless you have to)

The community cookbooks are robust and often, but not always, support a wide range of configuration options (like host name, app install directory).

BUT - sometimes (read: often) the community cookbook just isn't right. Go ahead and roll your own. Spin up a base Vagrant VM. Run some commands to install your software just the way you need. Then use those commands as the model for writing your cookbook with Chef resources, such as package, file, and service. Now you can reproduce your app install as many times as needed. OR - contribute a change back to the community cookbook via a GitHub Pull Request. We know how to do that.

Create our first cookbook

Generating your first cookbook starts with the chef generate cookbook <your_cookbook_name> command.

This will give you a directory structure with all files necessary stubbed out. Let's just dig in to the recipes/default.rb file and use it to install Apache 2.

# in recipes/default.rb
package 'apache2' do
  action :install
end

You've done it. Apache 2 will be installed when Chef Client runs. Congrats.

Test Kitchen

Very good. Now to test it. A few years ago, you'd approach testing like this: a) have a Chef Server handy, b) upload the cookbook to the server, c) install the chef-client on a Vagrant VM, d) register the Vagrant VM with the Chef Server, e) converge the node, and f) verify that your cookbook worked. Sounds like a pain in the butt and it was. With some Vagrant plugins the process was bearable, but still clunky.

But Flether Nichol decided we should have a better experience and instead created Test Kitchen. Similar to the joy that Mitchell Hashimoto delivered by putting VM settings in a Vagrantfile, Fletcher took Configuration Management testing to the next step and now we create .kitchen.yml files to describe our tests.

To test our cookbook, we'll run kitchen converge. It'll spin up your desired machine with your desired driver (Vagrant, Docker, AWS, OpenStack, etc), fake out a Chef Server with Chef Zero (or use a real Chef Server if that's your thing), and do a Chef client run of your cookbook.
When that completes (barring any errors in your cookbook), test the changes with kitchen verify. Or you can log in to the VM and check things out manually (kitchen login). All of which is driven by the data in your .kitchen.yml file.

Chefs to follow on Twitter

• Adam Jacob
• Michael Ducy
• Fletcher Nichol
• Chef

Podcasts to listen to

Food Fight Show    The Pre-Accident Investigations Podcast
Arrested DevOps   Software Defined Talk

Workout

We are going to install the Unifi controller software from Ubiquiti using the Chef community cookbook for Unifi.

1. From the top level of the Automation Tools git project vagrant up and vagrant ssh in to the big class VM that has everything installed.
2. Ensure the Chef DK is properly installed and configured by running chef verify from the command prompt.
3. Move in to the project's chef directory with a cd /vagrant/chef command.
4. Create a new cookbook with the chef generate cookbook bootcamp for our project.
5. As the resulting output says, we will want to cd bootcamp in to our new cookbook.
6. Edit the metadata.rb file to depend on the Unifi community cookbook from the Chef Supermarket by adding the line depends 'unifi' at the bottom of the file.
7. Edit the recipes/default.rb file to install Apache by including the default apache2 recipe include_recipe "unifi::default"
8. Also edit the recipes/default.rb file to create a user called bootcamp using the native user resource.

   user 'bootcamp' do
     action :create
   end

9. Make sure all the cookbook's dependencies, in this case the unifi cookbook is a dependency, have been downloaded by running the berks install command.
10. Run kitchen list to see what VMs Test Kitchen is going to create when testing your cookbook.
11. VMs in Vagrant are great, but compared to using containers in Docker the Vagrant VMs are mighty slow.
12. Avoid downloading Vagrant and entire Ubuntu (and CentOS) VMs by changing the .kitchen.yml file to have the contents of the kitchen.yml.example file already in the git project's chef directory.
13. Verify that you're going to be testing a single VM by running kitchen list again.
14. Converge the node with Test Kitchen kitchen converge. Note the output of the chef-client run.
15. Connect to the VM via SSH (kitchen login)
16. Verify your bootcamp user exists on the machine with an id bootcamp command.
17. Now we will check that Unifi is running with wget --no-check-certificate https://localhost:8443. Or by browsing to https://localhost:8443 in your own browser and port forwarding will bring the service up from the container inside the VM to be exposed on your own device.
18. Next up, testing that our VM is in the proper state using the kitchen verify command.
19. By default both of the tests will be skipped, it makes sense because Chef doesn't know what your cookbook will do.
20. Edit the tests/smoke/default/default_test.rb file so it will verify the bootcamp user exists and the test that port 8443 is listening will pass too (it { should be_listening }). Then run kitchen verify again.
21. When we're done with the VM you will exit and then the same as Vagrant, we will destroy it. kitchen destroy