wiki:Usage/UserDocumentation

Version 9 (modified by sil, 12 years ago) (diff)

Documented the datamodel of CMT

User Documentation

CMT can be used with different user interfaces:

Web Interface
An easy-to-use, self explainable, interface based on Django Admin. You only need a webbrowser to use this interface.
Command Line Interface (CLI)
A less-easy-to-use interface which could be used to make mass-updates. The CLI offers the user some powerful functions which aren't possible on the web-interface. To use this interface you need to install CMT on the computer you're using.

Please keep in mind: Both interfaces are lacking of an undo-function, so be careful when you're about to make any changes in the data you've stored in CMT.

About the CMT data(structure)

The datastructure of CMT consists of various models, defined in the models.py source-file. Just like in other Django projects, these models are used as a blueprint for objects in CMT, and for database tables to store these objects in. There's an interactive shell, included in Django, which you can use for testing queries or looking up the models. This shell has to be executed with the manage.py-file which is a wrapper around Django's command-line utility django-admin.py and can be found in the CMT-package directory. If you are familiar with Django, you might use this shell. Otherwise we advise you to use one one of the interfaces of CMT.

user@host$ python manage.py shell

This will give you an interactive shell with a welcome message. First thing to do is import the models of sara_cmt:

In [1]: from sara_cmt.cluster.models import *

Example of a model

To give an impression of how a model looks like, here's an example:

class Room(ModelExtension):
  address = models.ForeignKey(Address, related_name='rooms')

  floor   = models.IntegerField(max_length=2)
  label   = models.CharField(max_length=30, blank=False)

  class Meta:
    unique_together = ('address', 'floor', 'label')
    
  def __unicode__(self):
    return unicode(floor)

This model the class for a Room-model. In this case a room has got 3 fields of different types:

  • An address, which is stored in the database as a foreign key to the Address-table.
  • A floor, which is stored as an integer of 2 digits max.
  • A label, to give the room a human readable name. This label stored as a string of at least 1 character, to 30 characters max.

Available models

The models available in CMT are:

Model Description Related to
Cluster A labeled group of hardware pieces
HardwareUnit A specific piece of hardware Cluster (n:1)
Role (n:n)
Network (n:n)
HardwareModel (n:1)
WarrantyContract (n:1)
Rack (n:1)
Connection (n:1)
Interface An interface of a piece of hardware Network (n:1)
HardwareUnit (n:1)
InterfaceType (n:1)
Network Information about a network. Interface (1:n)
Rack A standardized system for mounting various pieces of hardware in a stack of slots.
Country Countries paired with there country-code
Address Holds information about the physical location of a piece of hardware. Country (n:1)
Room Located at an address, this is where a rack of hardware can be found.
Company This can be linked to hardware, to be able to define contactpersons for a specific piece of hardware Address (n:n)
Connection These are contacts that can be linked as connections for warranties etc. Address (n:1)
Company (n:1)
Telephonenumber Used for telephonenumbers of contacts. Country (n:1)
Connection (n:1)
HardwareModel This model is being used to specify some extra information (specifications) about a specific type (model) of hardware Company (n:1)
Role Describes the role of a piece of hardware in the cluster.
InterfaceType Contains information about different types of interfaces Company (n:1)
WarrantyType A type of warranty offered by a company Connection (n:1)
WarrantyContract A class which contains warranty information of (a collection of) hardware. WarrantyType (n:1)

Cluster

A labeled group of hardware units.

attributedatatypedescriptionexample
namestringhuman readable label of the cluster

required: name; unique: name

HardwareUnit

A specific kind of hardware.

attributedatatypedescriptionexample
clustermany-to-one relationshipcluster which this hardware is part of
rolemany-to-many relationshipthe role this hardware fulfills within the cluster
specificationsmany-to-one relationshiphardware specifications
warrantymany-to-one relationshipwarranty contract for this hardware
sellermany-to-one relationshipthe company which it's bought from
ownermany-to-one relationshipthe client/project which it's bought for
rackmany-to-one relationshiopthe rack this hardware is mounted in
first_slotintegerthe first slot this hardware occupies in its rack
statestring{new,clean,configured,unknown,off}, default: off (more)
warranty_tagstringservice tag or serialnumber
serial_numberstringserial number
labelstringlabel for the piece of hardware

required: cluster, role, rack; unique: rack+first_slot

Interface

An interface of a piece of hardware.

attributedatatypedescriptionexample
networkmany-to-one relationshipnetwork this interface is in
hostmany-to-one relationshippiece of hardware this interface is part of
iftypemany-to-one relationshiptype of interface
labelstringlabel for the interface
aliasesstringcomma-seperated list of CNAMEs
hwaddressstringMAC-address written with [0-9a-f:]
ipIP-addressIP-address

required: network, host, iftype, label; unique: network+ip, network+hwaddress

Network

Holds information about a network.

attributedatatypedescriptionexample
namestringhuman readable label.
netaddressIP-addressnetwork address192.168.1.0
netmaskIP-addressnetmask255.255.255.0
gatewayIP-addressgateway address. Automagically generated if kept empty.
domainstringdomain nameirc.sara.nl
vlanintegerVLAN139
hostnamesstringstringformat of hostnames in the networkib-{machine}

required: name, netaddress, netmask, domain, hostnames; unique: -

Rack

A cabinet used for mounting hardware.

attributedatatypedescriptionexample
roommany-to-one relationshipthe room where this rack is located
labelstringlabel for the rackam91
capacityintegernumber of slots

required: room, label, capacity; unique: room+label

Country

attributedatatypedescriptionexample
namestringname of the countryNetherlands
country_codeintegercountry code, according to E.164 standard31

required: name, country_code; unique: name, country_code

Address

Information about the physical locations, which can be referred to from objects.

attributedatatypedescriptionexample
countrymany-to-one relationshipcountry of the address
addressstringaddress-line
postalcodestringpostal code
citystringcity

required: address, city; unique: address+city

Room

A room is located at an address. This is where racks of hardware can be found.

attributedatatypedescriptionexample
addressmany-to-one relationshipaddress where the room is
floorintegerfloor of the room
labelstringlabel of the room

required: address, floor, label; unique: address+floor+label

Company

Holds information about a company. This can be used to link (to) hardware and contacts.

attributedatatypedescriptionexample
addressesmany-to-many relationshipaddresses where a company is located
namestringname of the company
websiteURLURL of their website

required: addresses, name, website; unique: name

Connection

A contact can be a person, or (for example) a hotline for support.

attributedatatypedescriptionexample
addressmany-to-one relationshipaddress of the contact
companymany-to-one relationshipcompany of the contact
activebooleanwhether it's currently a contact or not
namestringfull name
emailemail addressemail address

required: company, name; unique: company+name

Telephonenumber

Telephonenumber to link to a contact. Split in country-, area- and subscriber-part for easy filtering.

attributedatatypedescriptionexample
countrymany-to-one relationshipcountry part of the telephone number
connectionmany-to-one relationshipowner of the telephone number
areacodecharfieldarea code part of the telephone number
subscriber_numberintegerfieldsubscriber part of the telephone number
number_typecharfield['T','C','F'], for telephone, cellphone, or fax

required: country, connection, areacode, subscriber_number, number_type; unique: -

HardwareModel

This model is being used to specify some extra information (read: specifications) about a specific type of hardware.

attributedatatypedescriptionexample
vendormany-to-one relationshipmodel
vendorcodestringvendor's product codeCISCO7606-S
namestringhuman readable labelPowerEdge M610
rackspaceintegersize (number of slots) of the box
expansionsintegernumber of expansion slots

required: vendor, name, rackspace; unique: -

Role

These are possible roles for hardware in a cluster.

attributedatatypedescriptionexample
labelstringhuman readable labelcompute node

required: label; unique: label

InterfaceType

Contains information about different types of interfaces.

attributedatatypedescriptionexample
vendormanufacturer of the interface
labelstringhuman readable labelDRAC 5

required: label; unique: -

WarrantyType

A type of warranty offered by a company.

attributedatatypedescriptionexample
contactmany-to-one relationshipsupplier of the warranty
labelstringhuman readable labelDell Next Business Day On-Site

required: contact, label; unique: label

WarrantyContract

Contains warranty information of a specific contract, for (a collection of) hardware.

attributedatatypedescriptionexample
warranty_typemany-to-one relationshipthe type of warranty
contract_numberstringcontract numberNSEN420201
annual_costdecimalcost per year433.61
labelstringhuman readable labelLISA I vervanging
date_fromdatestart-date of the contract
date_todateexpiration-date of the contract

required: label, date_from, date_to; unique: label

Web-interface

The web-interface can be opened in any regular web-browser. Ask your administrator for its address. Once you open the web-interface, a login-page should appear where you have to issue your credentials.

From there you'll be able to add objects to the database, explore objects in the database, change objects in the database, or delete objects from the database. It's meant as a intuitive easy-to-use interface, with advanced Create-Read-Update-Delete (CRUD) functionalities. It makes you able to quickly explore the data stored in CMT, or act on the data. The most important entities you can act on are:

Clusters
A cluster could be described as a collection of hardware, connected to each other through one or more networks.
Racks
A rack is a cabinet that houses hardware like servers and switches. It has multiple slots to mount hardware. These slots identify the physical location of your hardware.
Hardware
Hardware units like (virtual) servers, switches, PDU's, etc. It could be 'any piece of hardware with a powercable attached to it'.
Networks
Networks defined by their (IP) address space. The addresses in the address space of a network each are related to an interface.
Interfaces
Network interfaces for input/output, thus any kind of card or (virtual) adapter which can have a network-cable plugged into it, and has a hardware address (like a MAC-address). Most obvious example is a regular 100/1000Mbit NIC, but it could be something like an InfiniBand interface card, or an out-of-band management adapter. Many network interfaces are related to a single piece of hardware, and a single network.

Command Line Interface

To use the Command Line Interface has to be installed on your system. If you issue the command cmt you'll get the usage info:

user@host$ cmt
Usage: cmt [options]

This is the commandline interface of the SARA package SARA CMT.

Options:
  -h, --help            show this help message and exit
  -n, --dry-run         This flag has to be given before -[aclmr]
  --script              
  -a ENTITY, --add=ENTITY
                        Add an object of the given ENTITY. Extra arguments
                        should be given like `set ASSIGNMENTS`, where
                        ASSIGNMENTS are formatted like `ATTR=VAL`
  -c ENTITY, --change=ENTITY
                        Change value(s) of object(s) of the given ENTITY.
                        Arguments should be given like `get QUERY set
                        ASSIGNMENTS`, where both QUERY and ASSIGNMENTS consist
                        out of one or more terms formatted like `ATTR=VAL`.
                        Only objects matching QUERY will be changed according
                        to the given ASSIGNMENTS
  -g TEMPLATE, --generate=TEMPLATE
                        Render the given TEMPLATE, and store the result if
                        specified
  -l ENTITY, --list=ENTITY
                        List object(s) of the given ENTITY. Arguments should
                        be given like `get QUERY`, where QUERY consists out of
                        one or more terms formatted like `ATTR=VAL`
  -r ENTITY, --remove=ENTITY
                        Remove object(s) of the given ENTITY. Arguments should
                        be given like `get QUERY`, where QUERY consists out of
                        one or more terms formatted like `ATTR=VAL`. Only
                        objects matching QUERY will be listed
  -v, --verbose         

More information about the options can be read in the following subsections. For each option we'll give one or more use cases as an example.

CRUD options

Just like the web-interface, the CLI can be used for CRUD-functionalities (Create, Read, Update, Delete) too. However, the implementation of the CLI is much more powerful, since it can be used to do things with multiple objects at once, and can be done from shell-scripts.

Add object: -a, --add

Use this option to add a new object of a given entity to the database. You have to specify the entity and assign values to its fields by providing assignments as extra arguments. The syntax is as follows:

cmt --add <entity> <assignments>

Use case

You want to add a new server, and its interfaces

As an example we'll add a server (PowerEdge M610) to the 1st position in rack 1 of the Lisa-cluster. The server will be used as a test- and computenode, and has the following interfaces:

Interface network MAC-address
Gigabit lisa admin 00:00:00:00:00:01
InfiniBand lisa infiniband 00:00:00:00:00:02
iDRAC6 lisa consoles 00:00:00:00:00:03
user@host$ cmt --add hardwareunit cluster=Lisa role=computenode role=testnode specifications='PowerEdge M610' rack=1 first_slot=1

user@host$ cmt --add interface host=r1n1 network='lisa admin' iftype='Gigabit' hwaddress='00:00:00:00:00:01'

user@host$ cmt --add interface host=r1n1 network='lisa infiniband' iftype='InfiniBand' hwaddress='00:00:00:00:00:02'

user@host$ cmt --add interface host=r1n1 network='lisa consoles' iftype='iDRAC6' hwaddress='00:00:00:00:00:03'

List object(s): -l, --list

Use this option to list existing objects in the database. You have to specify the entity, and filter objects you want to list by providing a query. The syntax is as follows:

cmt_sara --list <entity> [get] <query>

Use case

If you need to figure out where a machine -- in this case test3 -- is physically hosted:

When an engineer has to fix a node you need to tell him the location of the node. This information shall be listed by issuing the following command:

user@host$ cmt --list hardwareunit label=test3

The output of this command looks like:

 .---[  test3  ]---
 : cluster        : LDAP/CUA
 : address        : Amsterdam - Science Park 121
 : room           : S142A
 : rack           : rack AD38
 : first_slot     : 23
 : specifications : PowerEdge 1950 (Dell bv)
 : roles          : administration
 : in_support     : True
 '---

Change object(s): -c, --change

Use this option to change fields of one or more objects in the database. You have to specify the entity, filter objects you want to change by providing a query, and assign new values to their fields by providing assignments as extra arguments. The syntax is as follows:

sara_cmt --change <entity> [get] <query> set <assignments>

Use case

You have to change an existing address:

user@host$ cmt --change address get address='Kruislaan 415' set address='Science Park 121' postalcode='1098 XG'

Or maybe a motherboard has been replaced, so the MAC-addresses of the onboard interfaces have been changed:

Interface Old MAC New MAC
Gigabit 00:23:AE:FD:C9:00 00:26:B9:F9:95:34
iDRAC6 00:23:AE:FB:C7:DD A4:BA:DB:00:31:F6
user@host$ cmt --change interface get hwaddress=00:23:AE:FD:C9:00 set hwaddress=00:26:B9:F9:95:34

user@host$ cmt --change interface get hwaddress=00:23:AE:FB:C7:DD set hwaddress=A4:BA:DB:00:31:F6

In both commands you'll query for the interface with the old MAC-address, and overwrite the value of the hwaddress-field with the new MAC-address.

Since you know in which machine the motherboard has been replaced -- in this case r34n26 -- it might be easier to issue a command like:

user@host$ cmt --change interface get host=r34n26 iftype=Gigabit set hwaddress=00:26:B9:F9:95:34

user@host$ cmt --change interface get host=r34n26 iftype=iDRAC6 set hwaddress=A4:BA:DB:00:31:F6

This will only work when the machine contains a single Gigabit-interface, as well as a single iDRAC6-interface. In this case you don't need to know the old MAC-addresses.

You want to add the service-tags to new Dell-hardware:

For example new chassis in rack 2, 13, and 36:

user@host$ for rack in 2 13 36
> do
>   for cmc in a b
>   do
>     svctag=`sararac -c getsvctag dr-r${rack}cmc-${cmc} | sed -rn 's/^Chassis.*([A-Z0-9]{7})$/\1/p'`
>     echo saving warranty tag ${svctag} to r${rack}cmc-${cmc}
>     cmt --script --change hardwareunit get label=r${rack}cmc-${cmc} set warranty_tag=${svctag}
>   done
> done

And all hardware in these chassis:

user@host$ for rack in 2 13 36
> do
>   for node in `seq 1 32`
>   do
>     svctag=`sararac -c getsvctag dr-r${rack}n${node} | sed -rn 's/^.*([A-Z0-9]{7})$/\1/p'`
>     echo saving warranty tag ${svctag} to r${rack}n${node}
>     cmt --script --change hardwareunit get rack=${rack} first_slot=${node} set warranty_tag=${svctag}
>   done
> done

Remove object(s): -r, --remove

Use this option to remove objects from the database. You have to specify the entity, and filter objects you want to remove by providing a query. The syntax is as follows:

sara_cmt --remove <entity> [get] <query>

Use case

You have removed a server from a rack:

For example the first node in rack 2:

user@host$ cmt --remove hardwareunit rack=2 first_slot=1
INFO     remove 233 :: Found 1 objects matching query: r2n1
Are you sure? [Yn]

As you can see there will be asked for confirmation by default.

Other options

Dry-run: `-n, --dry-run

Use this option to suppress (CRUD-)actions that make changes to the database. Please note, this option has to be given before CRUD-options. So, use it like:

user@host$ cmt --dry-run --{add,list,change,remove} ENTITY ...

This option can become handy when you first want to test a command that'll have a big impact.

Use case

You're replacing all compute-nodes in a couple of racks (2-5), and want to test your command to remove the old hardware from CMT:

user@host$ cmt --dry-run --remove hardwareunit rack={2,3,4,5} role=computenode

Generate a config-file: -g, --generate

Render a template to generate a config-file or a report. Use the following syntax:

sara_cmt --generate <templatename>

By default your templates can be found in your (virtual) environment in de directory etc/cmt/templates.

Use case

When you want to generate the cfrun_hosts file of the Clearspeed cluster:

user@host$ cmt --generate cfrun_hosts.clearspeed

Which generates the cfrun_hosts file and stores the following content at /usr/sara/etc/cfengine/rc.hosts.clearspeed:

#
# This file is automagicly generated by CMT
#
#             SARA - Computing and Networking Services
# Date      : Tue, 15 Jun 2010
# Version   : CMT CLI 1.0
# Generated : Thu, 15 Jul 2010 09:20:27 +0200
#
# SVN:
#       $Id:$
#       $URL:$
#


outputdir=/data/cfengine/output/clearspeed
maxchild=8
login1
gb-r50n1.irc.sara.nl
gb-r50n2.irc.sara.nl
gb-r50n3.irc.sara.nl
gb-r50n4.irc.sara.nl
gb-r50n5.irc.sara.nl
gb-r50n6.irc.sara.nl
gb-r50n7.irc.sara.nl
gb-r50n8.irc.sara.nl
gb-r51n1.irc.sara.nl
gb-r51n2.irc.sara.nl
gb-r51n3.irc.sara.nl
gb-r51n4.irc.sara.nl
gb-r51n5.irc.sara.nl
gb-r51n6.irc.sara.nl
gb-r51n7.irc.sara.nl
gb-r51n8.irc.sara.nl

CMT Templates

CMT offers a powerful feature, powered by templates. With these templates you're able to let CMT generate various kind of files, like configuration files or reports about your clusters.

The syntax of the templates in CMT is based on Django templates, therefore the documentation of the Django template language can be useful as a reference. Django templates are usually meant for web pages, but since we use it for our configuration files some functionalities have been added for CMT-specific needs. By extending the Django template language, it has been made possible to:

  • Define an epilogue script
  • Pre-load datasets from database
  • Set a path to write the output to

To make use of these functionalities you have to start your template with

{% load cmt_client %}

and store the file in your template-directory as <name>.cmt

Epilogue script

Some configuration files need to be reloaded by the program when they are modified. Or maybe some other things should be done to activate the new configuration. This is where an epilogue script comes in. You can write the epilogue script in the template itself, between the epilogue and endepilogue tag:

{% epilogue %}
# this epilogue-script will restart the DHCP server
/etc/init.d/dhcp3-server restart
{% endepilogue %}

Pre-load datasets from database

You may define collections of objects to use in your templates. These collections can be defined with a filter, and assigning it to a variable name to use.

# syntax for the use-tag is:
#  use <entity> with <filter> as <variable name>
{% use network with 'name=lisa consoles' as net_consoles %}
{% use interface with 'network__name=lisa consoles' as ifaces_consoles %}

In the example two collections are assigned to a variable:

  • net_consoles will contain the network with the name lisa consoles
  • ifaces_consoles will contain all interfaces in that network

Now these collections can be used in the template, to build a part of a DHCP config-file for example:

group { # Start Console network
    option domain-name "{{net_consoles.domain}}";

    {% for iface in ifaces_consoles %}
    {% if iface.hwaddress != None %}
    host {{iface.fqdn}} {
        hardware ethernet {{iface.hwaddress}};
        fixed-address {{iface.ip}};
        option host-name "{{iface.fqdn}}";
    }
    {% endif %}
    {% endfor %}
} # End Console network

Set a path to write the output to

To save the result of your template, you can store the output to a file. This can be done by using the store-tag, and give it a path:

# syntax for the store-tag is:
#  store <outputpath>  as output
{% store /etc/dhcp/dhcpd.conf as output %}

In this example the result will be written to /etc/dhcp/dhcp.conf.