2013-11-28

Razor - What Lies Within the installer?

This is part #5 of the Razor series. In the last post we looked at Policies, Tags and touched slightly on Installers. In this post we will go more into detail into what exactly the installer does.

1. Installing Razor - Yes the New Version..
2. Razor - DHCP and TFTP
3. Installing the Razor client and creating a repository
4. Installers, Policies and Tags
5. Razor - What Lies Within the installer?
6. Installing ESXi with Razor

The installers are located in the /opt/razor-server/installers directory. In this directory you will find a yaml file and folder - both with the same name.

installer location

During this post - we will go through 2 separate installers, the Redhat one and the ESXi installer. The yaml file contains the following

redhat.yaml

The contents are quite self-explanatory, os_version, label, description and lastly the boot_sequence. The boot_sequence is the set of instructions that will be passed to the microkernel, the first time (1:) - boot to the installer (boot_install), and on the other boots (default:), boot to the local disk (boot_local) and bypass the microkernel.

The folder contains the following files (remember the os_version - therefore all the files are under the 6 subfolder).

redhat folder files

These files are the steps that will be executed, and as we saw in the redhat.yaml file the first boot will go to a boot_install.erb

boot_install

This file holds the correct boot parameters for booting into the installation - including the contents of the kernel_args template which we will see shortly. There is also information echoed out to the screen such as the Installer Label, the node's url, and the repository's url.

kernel_args

The kernel_args - has the pointer to the kickstart.erb file and the instruction to boot from the network device.

kickstart

The kickstart.erb file is mostly a standard kickstart file - with a few variables.

  • url=<%= repo_url %> - will be translated to the url for the repository.
  • rootpw <%= node.root_password %> - will be the password that you defined when you created the policy
  • network --hostname <%= node.hostname %> - will be the hostname according to the pattern that you defined in the policy.

After the rest of the standard kickstart contents (in the %post section), there is some more razor stuff.

kickstart - %post

  • %post --log=/var/log/razor.log - will log the rest of the process to /var/log/razor.log
  • curl -s -o /root/razor_postinstall.sh <%= file_url("post_install") %> - the node retrieves the post_install file and saves it to /root/razor_postinstall.sh
  • # Run razor_postinstall.sh on next boot via rc.local - this section is self explanatory.
  • curl -s <%= stage_done_url("kickstart") %> - here a API call is sent to razor to announce that kickstart is finished - and therefore the subsequent boots will go to local disk (as was defined in the redhat.yaml file)

post_install

The post_install.erb (the one that was saved to /root/razor_postinstall.sh) has a few more operations that are performed.

  • # Wait for network to come up when using NetworkManager - waits for the network to come up.
  • <%= render_template("set_hostname") %> - is the piece of code that sets the hostname.
  • <%= render_template("store_ip") %> - is the code that stores the IP address and reports back to razor server.
  • <%= render_template("os_complete") %> - sends an API call to the Razor server to notify that the OS installation is now complete.

The boot_local, set_hostname, store_ip and the os_complete pieces of code are located in the
/opt/razor-server/installers/common/ directory.

common directory
boot_local

The boot_local.erb has the instructions for the OS to boot to a local disk.

set_hostname

The set_hostname.erb has all regex magic that implants the hostname
(from the <%= node.hostname %> variable) into /etc/hosts.

store_ip

The store_ip.erb - retrieves the node's IP address and then reports back to the Razor server.

os_complete

The os_complete.erb updates the MOTD on the node, and removes the lines that razor added to the /etc/rc.d/rc.local file. It then goes back to the Razor server and invokes the broker.

If we were to look at the ESXi installation we would see something similar.

vmware_esxi.yaml

The contents of vmware_esxi.yaml file - the same format as before.

vmware_esxi directory

The contents of the vmware_esxi folder is slightly different.

boot_install

The boot_install.erb file here holds the instructions on how to boot the ESXi kernel - which is slightly different than that of a traditional Linux.

pxelinux_esxi

The pxelinux_esxi.cfg.erb is the file that was called from the previous step

  • KERNEL <%= repo_url('/mboot.c32') %> - the standard ESXi boot menu.
  • APPEND -c <%= file_url('boot.cfg') %> - add in the boot.cfg file.

boot.cfg

The boot.cfg.erb file modifies the boot string to point to the kickstart file on the Razor server.

ks.cfg

The ks.cfg.erb file has the kickstart instructions for ESXi - with great thanks to William Lam

  • rootpw --iscrypted <%= … %> - decodes the password that was defined in the policy.
  • wget <%= stage_done_url("kickstart") %> - here an API call is sent to the Razor server to notify that the kickstart process has completed and that the default boot process should now be used.

The ks.cfg file can be customized according to your needs of course.

A few things that need to emphasized about the current ESXi kickstart process - it has some limitations.

  1. The network configuration is DHCP only. There is no built in mechanism to accommodate static IP addresses at the present time. I will say that there is a method - which I will explain in a future post.
  2. Hostname configuration - is not covered either, that same future post will explain how to deal with this problem as well.
  3. There is no additional reporting on the node's state - besides the fact that kickstart has completed.

That is enough for this part. Next we will look at a full deployment.

2013-11-26

Razor - Installers, Policies and Tags

This is part 4 of a series of posts on the new version of razor-server

1. Installing Razor - Yes the New Version..
2. Razor - DHCP and TFTP
3. Installing the Razor client and creating a repository
4. Installers, Policies and Tags
5. Razor - What Lies Within the installer?
6. Installing ESXi with Razor

We are almost ready to deploy our first node but in order for that to happen - we will need the following.

  • Installer - the set of commands that will be run to deploy the OS
  • Broker - the next step for the installation of the node - usually the orchestration software
  • Policy - this will contain the repository name, the installer name
  • Hostname pattern - the naming policy for your deployed nodes
  • Password - the root password for the deployed node
  • Count - the maximum number of times you would like this policy to be applied
  • Tags - How to identify the node - based on the hardware reported. In the previous post we created a single repository - ESXi_5_1. The details of the repository are below.

    razor repos "ESXi_5_1"
    razor repos

    Create a broker. In our case - this will be the noop broker - which means that nothing will happen after the node has been provisioned.
    razor create-broker --name=noop --broker-type=noop
    razor brokers
    razor brokers
    Create a tag.
    Before we go into how the tags are created - I would like to explain in a bit more detail, what the tags are used for.
    The microkernel's main purpose is to scan the hardware of the node, and report that inventory back to the Razor server. With that information we can create categories/profiles/flavors of nodes - and deploy a operating to that node.
    Let's take an example. You have a UCS Chassis. You want to deploy ESXi on the Blades that have 128GB of RAM, but on the UCS blades that have 64GB of RAM you don't want ESXi, but rather RHEL6. You also have an HP Chassis with 16 blades, and there you want to deploy ESXi on all the servers.
    If we were to verbally describe the tags they would be as follows.
    Tag Rule
    UCS_ESXi Physical server + Manufacturer is Cisco + Blade has exactly 128GB of RAM
    UCS_Redhat Physical server + Manufacturer is Cisco + Blade has exactly 64GB of RAM
    HP_ESXi Physical server + Manufacturer is HP
    VM_Ubuntu Node is a VM + 1 vCPU + amount of RAM is <= 1GB
    For our example we will be creating a test rule and it's appropriate tag. The tag definitions will be:
  • 2 processors
  • It is a virtual machine
  • Greater/equal to 4GB ram

    cat > tag1.json << __CREATE_TAG_JSON___
    {
      "name": "Test_tag",
      "rule": ["and",
    ["=", ["num", ["fact", "processorcount"]], 2],
    ["=", ["fact", "is_virtual"], "true"],
    [">=", ["num", ["fact", "memorysize_mb"]], 4096]
       ]
    }
    __CREATE_TAG_JSON___
    razor create-tag --json tag1.json
    razor tags Test_tag

    razor create-tag

    Now to create the policy.


    cat > policy.json << __CREATE_POLICY_JSON___
    {
      "name": "ESXi_5_1",
      "repo": { "name": "ESXi_5_1" },
      "installer": { "name": "vmware_esxi" },
      "broker": { "name": "noop" },
      "enabled": true,
      "hostname": "host${id}.maishsk.local",
      "root_password": "blahblahbb5",
      "max_count": "100",
      "rule_number": "100",
      "tags": [{ "name": "Test_tag"}]
    }
    __CREATE_POLICY_JSON___

    The text above is piped to the policy.json file.
    create policy.json
    Let's go into a bit more detail into each of the lines of this policy.
    name The name of the policy
    repo The name of the repository that should be used to perform the deployment
    installer The name of the installer to be used - The name that should be used is the name of the yaml file (without the extension) which can be found in the /opt/razor-server/installers/ directory.

    image
    broker Which broker should be used to continue the configuration, in our case the noop broker - which means nothing will happen.
    enabled True or false - you can create a policy and not have it active - if you choose.
    hostname The pattern that will be used to name your nodes.
    host${id}.maishsk.local (id is the number that was allocated to the node) - the rest you can customize as you please.
    root_password The root password that will be configured on the node.
    max_count The number of times the policy should be applied.
    rule_number This allows you to prioritize your policies. Lower number wins
    tag The tag that the node will be matched against.

    razor create-policy --json policy.json
    razor policies ESXi_5_1

    razor create-policy

    The steps above were split into two stages, but there is also a possibility of creating the tag and the policy in the step, all you will need to do is define the rule in the policy.json - when the tag is declared.
    Instead of this line

    "tags": [{ "name": "Test_tag"}]

    You could do the exact same thing without creating the tag beforehand with the following syntax (all should be on one line)

    "tags": [ "name": "Test_tag", "rule": ["and",["=", ["num", ["fact", "processorcount"]], 2], ["=", ["fact", "is_virtual"], "true"], [">=", ["num", ["fact", "memorysize_mb"]], 4096]]]


    The difference in the syntax above is that we defined the rule of the tag - during the creation of the policy.
    That was a lot of information to digest in this post, so I hope you were able to get it all in.
    In the next post - we will go into a little more detail about the installers and their configuration.
  • 2013-11-18

    Installing the Razor-Client and Creating a Repository

    This is part three in the Razor Series. In part #2 we saw how to configure DHCP ad TFTP and as I mentioned in a previous post, the client is now a separate component.

    1. Installing Razor - Yes the New Version..
    2. Razor - DHCP and TFTP
    3. Installing the Razor client and creating a repository
    4. Installers, Policies and Tags
    5. Razor - What Lies Within the installer?
    6. Installing ESXi with Razor

    Here is how you would go about installing the razor-client which will allow you to interact with the server.
    I installed it in my $HOME folder

    First we will clone the code from Github.

    git clone https://github.com/puppetlabs/razor-client

    git clone

    Change directory to the razor-client folder and install the required gems

    cd razor-client
    bundle install

    gems

    Next create an alias that will include the url to the razor server (substitute the correct IP address / Hostname of course), and you should add this to your profile if you want the command to be available each time you login.

    alias razor="$PWD/bin/razor -u http://127.0.0.1:8080/api"

    create alias

    As you can see it is possible to install the client on the same machine as the server. If you are doing this then then you will not need the url - because it defaults to http://localhost:8080/api

    And if you run the razor command you should see the following output

    razor

    At the moment there is nothing there so looking at the repos/brokers/policies/nodes/tags will not give you much - yet…

    razor commands

    First some terminology.

    • Nodes - Nodes are the clients that connect into the system.
    • Brokers - Brokers are responsible for handing a node off to a config management system, like Puppet or Chef.
    • Tags - A tag consists of a unique name and a rule. The rule is a characteristic present on the the node
    • Repos - A repository where the installation files of an Operating System reside
    • Policies - This is a defined set of steps that tie all the other bits and pieces in Razor together, and are what gets ultimately applied to a node.
    • Installers - This will be the set of actions needed to deploy the operating system - you can think as a set of kick start instructions.
    Creating a repo, for that you will need to provide a URL to the installation ISO.
    razor create-repo --name=ESXi_5_1 --iso-url http://webserver/VMware-VMvisor-Installer-5.1.0.update01-1065491.x86_64.iso
    create-repo
    What currently happens is that razor will return an ID for the repo and begin the import process. The file is
    extracted to /tmp and eventually moved to the repo location that we defined in config file
    (/var/lib/razor/repo-store). A new directory will be created according to the repo name. 
    import files
    (Just as a side note - the repo is extracted into the /tmp folder but removed there after - so make sure you have adequate space on your razor server - or alternatively delete the temp files after the repo is created. There is currently a bug open on this issue.)
    When you now run razor repos you should see the repository you just created.

    razor repos

    razor repos

    In the next post we will see how we create a policy and see what we can do with tags.

    Razor - DHCP and TFTP

    This is part 2 of a series of posts on the new version of razor-server

    1. Installing Razor - Yes the New Version..
    2. Razor - DHCP and TFTP
    3. Installing the Razor client and creating a repository
    4. Installers, Policies and Tags
    5. Razor - What Lies Within the installer?
    6. Installing ESXi with Razor

    In the previous part we installed the razor-server component.

    But what use is razor without TFTP and DHCP servers that will allow you to boot your machines and install their OS? Because the architecture is modular, this can be done on different machines, or in this case - a single VM.

    Here is my layout of my environment.

    Environment

    My razor server has two NICs - one for access to the outside world and the other to provision my nodes. The reason I did it this way was because I do not control the allocation of IP addresses outside of my network and here I needed full control.

    Install dnsmasq

    apt-get install dnsmasq -y

    dnsmasq

    The configuration files of dnsmasq needs to modified.

    • The first 3 lines take care of the iPXE boot.
    • Enable the TFTP Server.
    • Root directory where the boot files will be located.
    • Set a DHCP range (this should be customized according to your environment).

    To add this basic configuration to the dnsmasq server you can use the following:

    cat >> /etc/dnsmasq.conf << __DNSMASQ_CONF__

    # This works for dnsmasq 2.45
    # iPXE sets option 175, mark it for network IPXEBOOT
    dhcp-match=IPXEBOOT,175
    dhcp-boot=net:IPXEBOOT,bootstrap.ipxe
    dhcp-boot=undionly.kpxe
    # TFTP setup
    enable-tftp
    tftp-root=/var/lib/tftpboot
    dhcp-range=192.168.1.50,192.168.1.150,12h

    __DNSMASQ_CONF__

    dnsmasq.conf

    The directory we defined above is not present by default - so it must be created, and then restart the dnsmasq service to start with the correct settings

    mkdir -p /var/lib/tftpboot
    service dnsmasq restart

    tftpboot directory

    Get the iPXE boot firmware and put it in the tftpboot directory..

    cd /tmp
    curl -L -O
    http://boot.ipxe.org/undionly.kpxe
    mv undionly.kpxe /var/lib/tftpboot/

    iPXE

    Create the razor iPXE bootstrap script. The nic_max parameter indicates the maximum number of NICs for which bootstrap.ipxe will report MAC's back to the razor server. In my case 3 was enough. File is generated on the fly according to the URL and parameters passed.
    (In order to eliminate problems with name resolution - it is easier to use an IP address instead of a hostname)

    curl -L -O http://192.168.1.5:8080/api/microkernel/bootstrap?nic_max=3
    mv bootstrap* /var/lib/tftpboot/bootstrap.ipxe

    bootstrap.ipxe

    Download and unpack the razor microkernel. This will create a microkernel folder under the root directory of your repo-store.

    curl -L -O http://links.puppetlabs.com/razor-microkernel-003.tar
    tar xf razor-microkernel-002.tar -C /var/lib/razor/repo-store/
    rm -rf razor-microkernel-002.tar

    microkernel

    microkernel files

    So just to recap. You should now have a DHCP/TFTP server with the correct configuration, iPXE, the razor bootstrap and also the razor microkernel. You are now ready to PXE boot your first node.

    When powering up a node and booting on the 192.168.1.0/24 network you should see something similar to the following.

    boot process

    Here you can see that the node received an IP, contacted the TFTP server, and started the bootstrap file. It then contacts the razor server, and starts to boot the microkernel.

    Within a short time you will have booted into the microkernel and should end up at this screen

    login

    How can you verify the that razor has recognized your node? Through the API.

    Looking at the collection of nodes - http://razor:8080/api/collections/nodes you should see the node you just brought up.

    node1

    What does razor know about the node? - http://razor:8080/api/collections/nodes/node1 

    node1 details

    Of course we could send API calls all day through a web browser, but that is not so convenient.

    In the next post we will install the razor-client and start create a installation repository.

    2013-11-13

    Installing Razor - Yes the New Version..

    This is part 1 of a series of posts on the new version of razor-server

    1. Installing Razor - Yes the New Version..
    2. Razor - DHCP and TFTP
    3. Installing the Razor client and creating a repository
    4. Installers, Policies and Tags
    5. Razor - What Lies Within the installer?
    6. Installing ESXi with Razor


    I assume that some of you are familiar with Razor - the provisioning system that was open sourced by EMC and written partially by Nick Weaver - his post about razor is here.

    Jonas Rosland (@virtualswede) created a great walkthrough on how to deploy a provisioning server, and get yourself up and running.

    In July 2013, the team decided on a new direction and a complete re-write of the product.

    Does that change anything? Yes! The instructions in the walkthroughs above do not work any more - since the core functionality has changed - substantially. This now a fully owned PuppetLabs project.

    When you run.. puppet module install puppetlabs-razor, that means you will get the new version of razor - not the previous version, and the steps above will not work. This first part will explain how to install the razor server.

    Most of the information in this post is also available on the Installation Wiki
    First the pre-requisites and caveats:
    • This was done on a freshly installed Ubuntu 12.04 VM. (The commands should
    • This is the way I got it to work, it does not necessarily mean that this is the optimal way.
    • I preferred to use the method of installing from Source and not the prepackaged modules (for a number of reasons).
    First we will install the necessary packages.

    apt-get update
    apt-get install postgresql libarchive-dev curl git openjdk-7-jre-headless -y


    Postgres is the database that is used, curl is not installed by default, git will be used for the source installation, libarchive is used to unpack the ISO images and we will need the JDK for some of the of the Ruby pieces

    Next we will prepare the database. Change the the following file /etc/postgresql/9.1/main/pg_hba.conf and add modify the file as below.

    # Database administrative login by Unix domain socket
    local all postgres peer

    # TYPE DATABASE USER ADDRESS METHOD
    host all all 127.0.0.1/32 md5

    # "local" is for Unix domain socket connections only
    local all all peer


    Changing context to the postgres user, create the user. You will be prompted for a password (twice).

    createuser -P -SDR razor



    Create the three databases, and go back to the root user.

    createdb -O razor razor_dev
    createdb -O razor razor_test
    createdb -O razor razor_prd
    exit


    create databases

    Restart the PostgreSQL service and check if the tables exist. You will be asked for the password for the razor user you created previously.

    service postgresql restart
    psql -h 127.0.0.1 -l -U razor razor_dev




    Here you see that there are 3 databases - owned by the razor user.
    Next we install rvm. We will use rvm to install the specific jruby version (1.7.4) that we are going to use. And source the file - so we can use it immediately.
    curl -L https://get.rvm.io | bash
    source /etc/profile.d/rvm.sh


    (Updated 14/11/13)

    After receiving some feedback from the razor guys - it was pointed out to me that rvm was the not the most ideal way to handle your ruby environment, rbenv is a much better solution. Here are the steps.

    curl
    https://raw.github.com/fesplugas/rbenv-installer/master/bin/rbenv-installer | bash

    rbenv

    And now to the correct bits to your top of your profile.

    # rbenv
    export RBENV_ROOT="${HOME}/.rbenv"
    if [ -d "${RBENV_ROOT}" ]; then
      export PATH="${RBENV_ROOT}/bin:${PATH}"
      eval "$(rbenv init –)"
    fi

    update .bashrc

    Reload the shell.

    source ~/.bashrc
    Install all the dependencies. This will download all the dependencies needed.
    rbenv bootstrap-ubuntu-12-04
    I found a gem that automatically runs rbenv rehash after each installation - which will save some time and worries.
    git clone https://github.com/sstephenson/rbenv-gem-rehash.git ~/.rbenv/plugins/rbenv-gem-rehash
    gem-rehash
    Next we install jruby-1.7.4rvm install jruby-1.7.4
    Install jruby 1.7.4 and set that as our global environment.

    rbenv install jruby-1.7.4
    rbenv rehash && rbenv global jruby-1.7.4

    jruby-1.7.4

    Install bundler.

    gem install bundler



    Clone the source code the local computer.

    cd /opt
    git clone https://github.com/puppetlabs/razor-server.git




    Retrieve the rest of the gems required. You will see a long list of gems downloaded and installed.

    cd razor-server/
    bundle install



    Create the config file and add in the necessary information. Here you will need the names of the databases you created before (razor_dev, razor_test, razor_prd) and the password for your razor user (in my case - razor).

    cp config.yaml.sample config.yaml
    nano config.yaml


    You will change the 3 connection strings in the file - like the example below. (please make sure you have the correct database names!!)



    Create a directory that will be the location of the installation ISO's.

    mkdir -p /var/lib/razor/repo-store

    Next we prepare the database.

    rake db:migrate



    And then prepare the web service for deployment, and start the process in the background.

    torquebox deploy
    torquebox run --bind-address 0.0.0.0 &


    You will then see a lot of output on the screen until you should get a line similar to the one below - showing that razor is now started.



    If all was done correctly - you should not have any errors in the output and we will verify the API is now available by going to http://razor:8080/api .



    If you get a json reply as above then your server is up and functioning.
    In the next post - we will see how to get to prepare the DHCP and TFTP parts of the solution.