In this post I follow on from Setting up Nova and Glance, and now moving installing and Integrating keystone. I’d first like to give credit to IBM developerWorks the guys in #openstack @ freenode IRC, and Psycle Interactive without whom I would not of been able to complete this write up.

Please be aware the following applies to 2011.3 ONLY! (Diablo Final) the configuration to come in Essex is far simpler, if when reading this post your packages are 2012.X you have just installed essex and this is not relevant, anyway here we go …

1

yum install openstack-keystone

Keystone itself has it’s own tirade of concepts to get to grips with … tenant, user, role, service, token etc … I’m not going to go into detail on those concetps, for that Please see the documentation.

Configuring mySQL

First thing I am going to do is change from sqlite to mySQL connection, this involves editing line 54 of /etc/keystone/keystone.conf

1

sql_connection = mysql://keystone:keystone@localhost/keystone

Ignoring the default_store configuration at the top of the file, as this states sqllite, from what I can tell this simply instructs keystone to use the sqlAlchemy driver, which we just updated to point to mySQL.

Now like glance we need to restart keystone for the database to be populated.

1

service openstack-keystone restart

Now run keystone-manage with no args if you see

1
2
3

File "/usr/lib/python2.6/site-packages/keystone/manage/__init__.py", line 283, in main
    raise exc
sqlalchemy.exc.OperationalError: (OperationalError) (1044, "Access denied for user 'keystone'@'localhost' to database 'keystone'") None None

Review your keystone.conf file and ensure your mySQL credentials are correct, once done start keystone again.

Initial Credentials

Now we need to create an admin Tenant, and add an admin user to this tenancy.

1
2
3
4
5
6
7
8
9
10

keystone-manage tenant add adminTenant
SUCCESS: Tenant adminTenant created.
keystone-manage user add adminUser <password>
SUCCESS: User adminUser created.
keystone-manage role add Admin
SUCCESS: Role Admin created successfully.
keystone-manage role grant Admin adminUser
SUCCESS: Granted Admin the adminUser role on None.
keystone-manage role grant Admin adminUser adminTenant
SUCCESS: Granted Admin the adminUser role on adminTenant.

Ok so we have just:

1. setup a tenant named adminTenant.
2. setup a user named adminUser and specified their password.
3. created an admin role.
4. assigned the adminUser to the Admin role.
5. granted adminUser the Admin role to the adminTenant

Note: the outputs are a little confusion on the role assignments…

“Granted Admin the adminUser role on adminTenant”,

it appears the string output has the arguments in the wrong order here it should read:

“Granted adminUser the Admin role on adminTenant”.

I have however verified the mySQL data and can see the roles being correctly assigned.

Also the output from

1
2

keystone-manage role grant help
Missing arguments: role grant 'role' 'user' 'tenant (optional)'

Confirms the arguments are being entered in the correct order.

i.e.

mysql> select * from user_roles;
+----+---------+---------+-----------+
| id | user_id | role_id | tenant_id |
+----+---------+---------+-----------+
|  1 |       1 |       1 |      NULL |
|  2 |       1 |       1 |         1 |
+----+---------+---------+-----------+
2 rows in set (0.00 sec)

Now we need to configure keystone to recognise these new admin roles.

Lines 41 and 44:

1
2

keystone-admin-role = Admin
keystone-service-admin-role = KeystoneServiceAdmin

Edit these to reflect your Admin role accordingly and then restart openstack-keystone The above shows seperate roles for general and service admin, in my case I set these to the same role, it is of course entirely up to you and your delegation setup. If you choose to retain the KeystoneServiceAdmin delegation you will need to setup the role as per the Admin role above and run through the grants accordingly.

Setting up the Service token and service definitions

1
2

keystone-manage token add 999888777666 adminUser adminTenant 2012-12-23T00:00
SUCCESS: Token 999888777666 created.

If instead you get an error:

1
2
3
4
5
6
7
8

ERROR: 'NoneType' object has no attribute 'id'
2012-04-23 12:27:29    ERROR [root] 'NoneType' object has no attribute 'id'
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 16, in <module>
    keystone.manage.main()
  File "/usr/lib/python2.6/site-packages/keystone/manage/__init__.py", line 283, in main
    raise exc
AttributeError: 'NoneType' object has no attribute 'id'

check your have correctly entered adminUser adminTenant (or the details you have entered) including correct capitilization.

1
2
3
4
5
6

keystone-manage service add nova compute "Openstack Compute Service"
SUCCESS: Service nova created successfully.
keystone-manage service add glance image "Openstack Image Service"
SUCCESS: Service glance created successfully.
keystone-manage service add keystone identity "Openstack Image Service"
SUCCESS: Service keystone created successfully.

Defining endPoints

Nova Here I managed to confuse myself, so let me be clear, this needs the nova_api service ip, not each compute node, meaning you only need one endpoint.

1
2

keystone-manage endpointTemplates add regionOne nova http://<nova_api_ip>:8774/v1.1/%tenant_id% http://<nova_api_ip>:8774/v1.1/%tenant_id% http://<nova_api_ip>:8774/v1.1/%tenant_id% 1 1
SUCCESS: Created EndpointTemplates for nova pointing to http://<nova_api_ip>:8774/v1.1/%tenant_id%

The 3 URL arguments are for publicURL, internalURL, adminURL (No idea if that is the order).

Glance

1
2

keystone-manage endpointTemplates add regionOne nova http://<glance_ip>:9292/v1 http://<nova_api_ip>:9292/v1 http://<nova_api_ip>:9292/v1 1 1
SUCCESS: Created EndpointTemplates for glance pointing to http://<glance_ip>:9292/v1

Keystone

1
2

keystone-manage endpointTemplates add pi-whc keystone http://<keystone_ip>:5000/v2.0 http://<keystone_ip>:5000/v2.0 http://<keystone_ip>:5000/v2.0 1 1
SUCCESS: Created EndpointTemplates for keystone pointing to http://<keystone_ip>:5000/v2.0.

Configuring Nova

Now we have keystone setup we need to configure nova to use keystone for authentication, by editing /etc/nova/api-paste.ini. Now there are seveal edits required, as such what follows are snippets of those changes.

EC2 Section modification

line 22 and 27 ([pipeline:ec2cloud] and [pipeline:ec2admin] sections).

1

pipeline = logrequest totoken authtoken keystonecontext ec2noauth cloudrequest authorizer ec2executor

New section for EC2 (in my config lines 60-61)

1
2

[filter:totoken]
paste.filter_factory = keystone.middleware.ec2_token:EC2Token.factory

Openstack section modification

Modification to [pipeline:openstackapi10] and [pipeline:openstackapi11] sections.

1
2
3
4
5

[pipeline:openstackapi10]
pipeline = faultwrap authtoken keystonecontext ratelimit extensions osapiapp10

[pipeline:openstackapi11]
pipeline = faultwrap authtoken keystonecontext ratelimit extensions osapiapp11

Shared section addition

We now need to add a complete new subsection to the .ini file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = keystone.middleware.nova_keystone_context:NovaKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

NOTE: you will want to change this to https, but I will not be covering https configuration in this post.

Check that your configuration is working:

1

curl -d '{"auth": {"tenantName": "adminTenant", "passwordCredentials":{"username": "adminUser", "password": "password"}}}' -H "Content-type: application/json" http://<keystone_ip>:35357/v2.0/tokens | python -mjson.tool

Now restart openstack-nova-api

1

service openstack-nova-api restart

Verifying nova keystone integration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

nova --debug --username=adminUser --apikey=<password> --url=http://<keystone_ip>:5000/v2.0 --version=1.1 list
connect: (<keystone_ip>, 5000)
send: 'POST /tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 69\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"passwordCredentials": {"username": "adminUser", "password": "<password>"}}'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 60
header: Date: Mon, 23 Apr 2012 14:16:13 GMT
Traceback (most recent call last):
  File "/usr/bin/nova", line 9, in <module>
    load_entry_point('python-novaclient==2.6.1', 'console_scripts', 'nova')()
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 209, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 166, in main
    self.cs.authenticate()
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/client.py", line 54, in authenticate
    self.client.authenticate()
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 140, in authenticate
    auth_url = self._v2_auth(auth_url)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 180, in _v2_auth
    resp, body = self.request(token_url, "POST", body=body)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 87, in request
    raise exceptions.from_response(resp, body)
novaclient.exceptions.BadRequest: Expecting auth (HTTP 400)

Don’t PANIC! it seems there was never a 2011.3 build for python-novaclient, as such we can “cheat” a little, and use 2012.1-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

rpm -Uvh http://pbrady.fedorapeople.org/openstack-el6/python-novaclient-2012.1-1.el6.noarch.rpm
nova --debug --os_username=adminUser --os_password=<password> --os_tenant_name=adminTenant --os_auth_url=http://<keystone_ip>:5000/v2.0/ usage-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "psycle"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:14:00 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/os-simple-tenant-usage?start=2012-03-26T16:14:00.749451&end=2012-04-24T16:14:00.749491&detailed=1 HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 21
header: Date: Mon, 23 Apr 2012 15:14:00 GMT
Usage from 2012-03-26 to 2012-04-24:
+-----------+-----------+--------------+-----------+---------------+
| Tenant ID | Instances | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+-----------+-----------+--------------+-----------+---------------+
+-----------+-----------+--------------+-----------+---------------+

You can also follow diablo more closely by using griddynamics’ rpm package

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

rpm -e --nodeps python-novavclient
rpm -Uvh http://yum.griddynamics.net/yum/diablo/python-novaclient-2011.3-b2489.noarch.rpm
nova --debug --username adminUser --password <password> --tenant_name adminTenant --auth_url http://<keystone_ip>:5000/v2.0/ usage-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "<password>"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:27:01 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/os-simple-tenant-usage?start=2012-03-26T16:27:01.859467&end=2012-04-24T16:27:01.859524&detailed=1 HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 21
header: Date: Mon, 23 Apr 2012 15:27:01 GMT
Usage from 2012-03-26 to 2012-04-24:
+-----------+-----------+--------------+-----------+---------------+
| Tenant ID | Instances | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+-----------+-----------+--------------+-----------+---------------+
+-----------+-----------+--------------+-----------+---------------+

BE WARNED

Most of the other commands for myself are presently returning 404 / 500 errors, with the Essex Release Impending the current EPEL advice seems to be to use Essex, I will update as/when I can with futher information on these issues.

For instance on a: flavor-create a 500 error is encountered with the following logged in api.log

1
2
3

...
(nova.api.openstack): TRACE: AttributeError: 'ControllerV11' object has no attribute 'create'
...

Configuring Glance

Modify /etc/glance/glance-api.conf

Comment out line 138 and uncomment 140

1
2
3
4

[pipeline:glance-api]
#pipeline = versionnegotiation context apiv1app
# NOTE: use the following pipeline for keystone
pipeline = versionnegotiation authtoken auth-context apiv1app

Modify lines 165-174 accordingly

1
2
3
4
5
6
7
8
9
10

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

now edit /etc/glance/glance-registry.conf and again comment out the current pipline= line and uncomment the keystone line.

1
2
3
4

[pipeline:glance-registry]
#pipeline = context registryapp
# NOTE: use the following pipeline for keystone
pipeline = authtoken auth-context registryapp

Update the authtoken filter accordingly

1
2
3
4
5
6
7
8
9
10

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

Restart glance

1
2
3
4
5

for i in api registry; do service openstack-glance-$i restart; done
Stopping openstack-glance-api:                             [  OK  ]
Starting openstack-glance-api:                             [  OK  ]
Stopping openstack-glance-registry:                        [  OK  ]
Starting openstack-glance-registry:                        [  OK  ]

testing Keystone

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

nova --debug --username adminUser --password <password> --tenant_name adminTenant --auth_url http://<keystone_ip>:5000/v2.0/ image-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "<password>"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:48:56 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/images/detail HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 14
header: Date: Mon, 23 Apr 2012 15:48:56 GMT
+----+------+--------+--------+
| ID | Name | Status | Server |
+----+------+--------+--------+
+----+------+--------+--------+

More to follow soon as I work through these issues, and later move onto 2012.X (Essex)

Now this is not to say that git is infallable. At any rate here is the best way to use git as an svn client and still maintain access to branches and tags.

git svn init -s <protocol>://<FQDN of server>/<repo path> ./folder_to_checkout_to

OR

git svn init -t tags -b branches -T trunk <protocol>://<FQDN of server>/<repo path> ./folder_to_checkout_to

1. This initializes an empty git respository in the folder specified (This can also be handy for migrating from subversion)
2. The examples above use -s for stdlayout, the 2nd example can be used to specify exact locations of trunk,tags,branches.

Now we need the data:

git svn fetch

One thing to note that whilst tags are present within git, subversion tags do not (at this stage) translate into git tags, as such to checkout svn tags (if you are not yet ready to make the jump to using git and want to maintain a subversion server this will needed).

git checkout -b tags/tag_name tags/tag_name

What does this do? it will checkout the tag from subversion: tag_name and setup a local git branch to track it.

If you are ready however to make the jump to git you can http://gitready.com/advanced/2009/02/16/convert-git-svn-tag-branches-to-real-tags.html.

1. You have a mysql instance installed and running
2. You have a rabbitmq-server installed and running
3. You have kvm installed and running (libvirt)
4. You have selinux set to permissive, as I will not be covering selinux rules here at this time and I do not think disabled is a valid option ;-)

I will also be carrying out mySQL configuration of glance and nova, for 2011.3 (Diablo), though most if not all of this should be portable to the Essex release

Install EPEL

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Install Nova and Glance

yum -y install openstack-nova openstack-glance

yum should take care of all the dependencies here, and install both with a minimal configuration.

Burning and Rebuilding bridges

First thing’s first KVM is going to install with it’s own default bridged networking, this provides NAT.

Which is also noted as being very slow (There is/was an note on the wiki@ linux-kvm.org but I have been unable to locate it at the time of writing)

If you are only setting this up for experimentation you can run with the default networking, simply use vibr0 in your nova.conf instead of br0, and ensure you have ipv4 forwarding enabled.

Burning Bridges

1
2
3
4
5
6
7
8
9

virsh net-list
Name                 State      Autostart
-----------------------------------------
default              active     yes 
virsh net-destroy default
Network default destroyed
virsh net-undefine default
Network default has been undefined
service libvirtd restart

Building Bridges

The theory here is that this configuration of bridge will give us near native network performance, which if you are setting up for use beyond a throwaway sandbox, you really do not want to start introducing bottlenecks.

Shutdown and disable NetworkManager

1
2
3

service NetworkManager stop
chkconfig NetworkManager off
chkconfig network on

If you know of a NetworkManager friendly way of doing the following please let me know!

In this scenario br0 becomes your current eth0

/etc/sysconfig/network-scripts/ifcfg-br0

1
2
3
4
5
6
7
8

DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.99.1
NETMASK=255.255.255.0
GATEWAY=192.168.99.254
ONBOOT=yes
DELAY=0

/etc/sysconfig/network-scripts/ifcfg-eth0

1
2
3
4
5
6
7

DEVICE=eth0
BOOTPROTO=none
TYPE=Ethernet
HWADDR=00:11:22:33:44:55
ONBOOT=yes
USERCTL=no
BRIDGE=br0

There is plenty more fun to be had here such as bonded interfaces (I myself have a few systems with bonded interfaces as such becoming br0 -> bond0 -> NIC’s), but that’s for another time.

Note: you may also use brctl for temporary configurations if you are just experimenting.

Caution: my network dropped out immediatly on my testbox, most likely because networkmanager was running, always ensure you can attach to the head of your box when doing network configuration ;-)

Once you have these configurations in place (Ensuring your have replaced the placeholder IP’s and MAC address with valid ones) you can now go for a

1

service network restart

All being well you’ll lose and re-establish connection, of you’ll be attaching a monitor / to kvm over ip.

Configuring Nova

First we’re going to need a blank database, please ensure you change the placeholder password that follows for something more secure, and amend the host if you are using mySQL on the same host as nova.

create database nova;
grant all privileges on nova.* to 'nova'@'localhost' identified by 'nova';

Your /etc/nova.conf should resemble this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lib/nova/tmp
--dhcpbridge=/usr/bin/nova-dhcpbridge
--dhcpbridge_flagfile=/etc/nova/nova.conf
--injected_network_template=/usr/share/nova/interfaces.template
--libvirt_xml_template=/usr/share/nova/libvirt.xml.template
--vpn_client_template=/usr/share/nova/client.ovpn.template
--credentials_template=/usr/share/nova/novarc.template
--network_manager=nova.network.manager.FlatDHCPManager
--iscsi_helper=tgtadm
--sql_connection=mysql://nova:nova@localhost/nova
--rabbit_host=localhost
--glance_api_servers=localhost:9292
--iscsi_ip_prefix=10.0.0.1
--bridge=br0

Setup the database and start the relevant nova services

nova-manage db sync
for i in api network scheduler compute; do service openstack-nova-$i start; done
for i in api network scheduler compute; do chkconfig openstack-nova-$i on; done

Note: you could also use openstack-nova-db-setup instead of “nova-manage db sync”, but it requires mysql-server, which at the time of writing if you have Percona installed will falsely adivse you a need to install mysql-server, Percona need to add: “Provides: mysql-server” to their spec ideally.

Remember this is only a basic setup so a lot of the options are left default such as the network_manager, I will cover their options at a later date.

Onto setting up a basic user (Note: this will be replaced in future posts with keystone)

1
2
3

nova-manage user admin saiweb
nova-manage project create saiweb saiweb
nova-manage network create saiweb 192.168.99.1/24 1 256 --bridge=br0

Take a moment to run a quick check on your services and network

1
2
3
4
5
6
7
8
9

nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-network     oneiroi                              nova             enabled    :-)   2012-03-07 22:21:10
nova-compute     oneiroi                              nova             enabled    :-)   2012-03-07 22:21:12
nova-scheduler   oneiroi                              nova             enabled    :-)   2012-03-07 22:21:10

nova-manage network list
id      IPv4                IPv6            start address   DNS1            DNS2            VlanID          project         uuid           
1       10.0.0.0/24         None            10.0.0.2        8.8.4.4         None            None            None            7d480f13-47f7-4117-9889-d44f378c3fee

Now we need the nova credentials for this user + project.

1
2
3
4
5
6
7
8
9
10

nova-manage project zipfile saiweb saiweb
unzip nova.zip
mv ./{novarc,pk.pem,cert.pem,cacert.pem} ~/.nova/
chmod 700 ~/.nova
chmod 600 ~/.nova/*
rm ./nova.zip
echo ". ~/.nova/novarc" >> ~/.bashrc
source ~/.bashrc
euca-add-keypair nova_key > ~/.nova/nova_key.priv
chmod 600  ~/.nova/nova_key.priv

Configuring Glance

The only change I made was to make glance use mysql.

create database glance;
grant all privilges on glance.* to 'glance'@'localhost' identified by 'glance';

/etc/glance/glance-resgistry.conf

1
2
3

...
sql_connection = mysql://glance:glance@localhost/glance
...

Once you have made the change, unlike nova all you need do is start glance and it will setup the database.

for i in api registry; do chkconfig openstack-glance-$i on; service openstack-glance-$i start; done

Now were going to need an image, I’m using the BT5-R2 .iso as an example, you could use any of the pre-generated images out there, or even build them using oz

1

glance add name="BT5-R2-Gnome-x64" is_public=True container_format=ovf disk_format=raw < ./BT5R2-GNOME-64.iso

Once the import has completed it should appear in your glance index

1
2
3
4

glance index
ID               Name                           Disk Format          Container Format     Size          
---------------- ------------------------------ -------------------- -------------------- --------------
1                BT5-R2-Gnome-x64               raw                  ovf                      2762084352

And assuming you setup your nova.conf correctly you should now be able to see this image from nova

1
2
3
4
5
6

nova image-list
+----+------------------+--------+
| ID |       Name       | Status |
+----+------------------+--------+
| 1  | BT5-R2-Gnome-x64 | ACTIVE |
+----+------------------+--------+

You will also have some default instance sizes aka flavours (commands use american spelling flavor).

1
2
3
4
5
6

nova-manage flavor list
m1.medium: Memory: 4096MB, VCPUS: 2, Storage: 40GB, FlavorID: 3, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.large: Memory: 8192MB, VCPUS: 4, Storage: 80GB, FlavorID: 4, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.tiny: Memory: 512MB, VCPUS: 1, Storage: 0GB, FlavorID: 1, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.xlarge: Memory: 16384MB, VCPUS: 8, Storage: 160GB, FlavorID: 5, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.small: Memory: 2048MB, VCPUS: 1, Storage: 20GB, FlavorID: 2, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB

Booting your first Instance

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

nova boot --flavor 2 --image 1 "BT5"
+--------------+--------------------------------------+
|   Property   |                Value                 |
+--------------+--------------------------------------+
| accessIPv4   |                                      |
| accessIPv6   |                                      |
| adminPass    | pnFKeVPpbb7bKKy6                     |
| config_drive |                                      |
| created      | 2012-03-07T23:11:59Z                 |
| flavor       | m1.small                             |
| hostId       |                                      |
| id           | 1                                    |
| image        | BT5-R2-Gnome-x64                     |
| key_name     | None                                 |
| metadata     | {}                                   |
| name         | BT5                                  |
| progress     | 0                                    |
| status       | BUILD                                |
| tenant_id    | saiweb                               |
| updated      | 2012-03-07T23:11:59Z                 |
| user_id      | saiweb                               |
| uuid         | fb08be47-2647-4cb2-86d8-867ea0ef4981 |
+--------------+--------------------------------------+
virsh list
 Id Name                 State
----------------------------------
  1 instance-00000001    running

And as iso-boot is not currently complete, this example falls down here, as the instance fails to boot from the .iso file, still you now have

1. Successfully configured nova
2. Sucessfully configured glance
3. Have nova using glance

All you need do is load a valid image into glance and boot using nova, so now I will be cheating a little I will create a blank 10GB qcow2 image, import it into glance boot it and use virt-manager to attach the .iso and reboot.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

qemu-img create -f qcow2 blank.qcow2 10G
Formatting 'blank.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536
glance add name="blank-10G" is_public=True container_format=bare disk_format=qcow2 < ./blank.qcow2
Added new image with ID: 2
nova boot --flavor 2 --image 2 "BT5"
+--------------+--------------------------------------+
|   Property   |                Value                 |
+--------------+--------------------------------------+
| accessIPv4   |                                      |
| accessIPv6   |                                      |
| adminPass    | H3khDYMwheNNWBV3                     |
| config_drive |                                      |
| created      | 2012-03-07T23:01:50Z                 |
| flavor       | m1.small                             |
| hostId       |                                      |
| id           | 2                                    |
| image        | blank-10G                            |
| key_name     | None                                 |
| metadata     | {}                                   |
| name         | BT5                                  |
| progress     | 0                                    |
| status       | BUILD                                |
| tenant_id    | home                                 |
| updated      | 2012-03-07T23:01:50Z                 |
| user_id      | oneiroi                              |
| uuid         | 05ce2b5d-d03c-442e-99e3-2c079469ec5b |
+--------------+--------------------------------------+

Now I cheat I used virt-manager to force off the insance, create and attach an IDE cdrom and set it as the primary boot device. BT5 boots from the ISO and I can even begin to work through the install to hard drive menus, which as irony would have it prompts me that it needs an 11.5GB partition to install upon :D

I will cover producing proper images in my next openstack post, as the size of the storage volume should not be defined by the image in glance, it should be defined by the falvour being started.

n2n is a layer-two peer-to-peer virtual private network (VPN) which allows users to exploit features typical of P2P applications at network instead of application level. This means that users can gain native IP visibility (e.g. two PCs belonging to the same n2n network can ping each other) and be reachable with the same network IP address regardless of the network where they currently belong. In a nutshell, as OpenVPN moved SSL from application (e.g. used to implement the HTTPS protocol) to network protocol, n2n moves P2P from application to network level.

So why do I care ?

Some services you may wish to run on a public cloud such as Gluster do not have (at the time of writing) internal TLS (read: encryption), this n2n allows you to establish peer to peer vpn connections, wihtout the need of a single routing device (with some assume caveats I will cover shortly).

So in short you can have your own private network within the cloud environment without affecting that environment, this allows for:

1. TLS for services otherwise sent “in the clear”
2. Potential for Cluster services and floating IP’s without touching the host network infrastructure.

Installation

We are going to use EPEL, why? because I’m a packager and I will be using redhat for this setup, so admitedly I am a little biased toward RedHat, that said the majority of the following configurations should be portable to other distros, leave a commment if you get stuck I will try to help!

yum -y install n2n

And no I’m not using sudo i.m.o sudo is akin to “training wheels”, and somethign I will only generally use if I have too (such as maintaining an auditable system), you are of course welcome to use sudo yourself, I use “throw away” vm’s for all my experimentation so in these cases the ethos is if it’s broken it gets rebuilt.

SuperNode Setup

First thing’s first we’re going to need at least 1 Supernode, as I uderstand it a Supernode is used to register new peers and to retrieve currently connected peers. Once this list is retrieved the individual nodes will communicate directly (p2p), and not via the supernode.

Caveats to note:

1. If all supernodes are down, only existing peers can communicate, new peers can not.

supernode whilst installed does not at the time of writing provide an init.d/sysvinit script, you may use the following:

place the above in /etc/init.d/supernode and chmod +x i.e.

curl -o /etc/init.d/supernode https://raw.github.com/gist/1986260/b66b38da265ea14aac8d0ef7196a9ba98939716c/supernode.sh && chmod +x /etc/init.d/supernode

(Though I really do recommend you read through this code first before trusting it blindly!)

Note: Annoyingly I had to use the -f (foreground) flag to allow the daemon wrapper to function correctly with this process, there is more than likely a better solution, please feel free to revise the gist it is public.

Now as I have opted to use a non existant n2n account to daemonize the process this will need creating as will the pid directory.

useradd -d /dev/null -s /sbin/nologin n2n
mkdir /var/run/supernode && chown n2n:n2n /var/run/supernode

You will now be able to start your supernode with: /etc/init.d/supernode start.

In my configuration above I have chosen to bind port 1200, you can change this to any port, but remember that your vpn peers will need to be able to access this port. As such you will need the relevant iptables rules

iptables -N N2N
iptables -I INPUT -j N2N
iptables -A N2N -s <vpn peer> -p udp --dport 1200 -j ACCEPT

I highly recomend you limit your firewall to only allow connection from known peers, and that this is done over the internal interface (for which you do not generally pay bandwidth charges).

I also recomend you repeat this process on a 2nd node to provide 2 Supernodes (The maximum allowable) for greater resilliance.

Edge Setup

I have opted for a .conf file approach here, you can of course opt to instead embed everything in the sysvinit script.

DEVICE="n0"
ADDRESS="127.16.0.1"
MAC="00:11:22:33:44:55"
COMMUNITY="N2N"
SHAREDKEY="asdf12345"
SUPER1="1.2.3.4:1200"
SUPER2="1.2.3.5:1200"
PORT="1201"

Place this in /etc/edge.conf, you can negate ADDRESS if you wish to use DHCP, whilst you can also Negate SUPERNODE2 and MAC I do not recomend doing so for the following reasons.

1. Negating Supernode2 means there is only 1 supernode and as such a single point of failiure in the setup
2. Negating MAC is valid, however on loss of connection and restoration a new MAC is generated meaning all existing nodes can not communicate with the restored node untill their local ARP caches are cleared, specifiying a static MAC address ensures immediate restoration of communication.
3. I have made PORT a requirement, it is technically optional but fixing the port makes your iptables / firewall rules far easier.
4. Make sure you actually edit the file and replace the args with VALID ones, especially the SHAREDKEY as the above is in no way secure!
5. Make sure your ip and mac addresses are unique!

We need to prep the pid dir again:

mkdir /var/run/edge && chown n2n:n2n /var/run/edge

place the above in /etc/init.d/edge and chmod +x i.e.

curl -o /etc/init.d/edge https://raw.github.com/gist/1986260/3061d0fb9d6f2ddf1608f01917129d65b8131d33/edge.sh && chmod +x /etc/init.d/edge

(Again I HIGHLY recommend you actually read the code before blindly trusting it!)

Note: the –user option is negated in this init file. This is because we need to actually create a network interface, something that can only be done as root. As such we are reliant on the edge binary to drop privileges itself by providing the -u and -g arguments, these are of course assuming you have allready setup the n2n user, as per above and not just skipped to this section.

Add the Services and set them to run

chkconfig --add supernode
chkconfig --add edge
chkconfig supernode on
chkconfig edge on

Modify other services that are reliant on the VPN

Modify the “Requires” line in the sysvinit script for each service you want to only start once your VPN has been established.

...
# Required-Start: $local_fs $network $supernode $edge
...

Note: Whilst I have opted for requiring supernode here, you do not need this, you can require just your edge service, as the supernode does not have to run on the same device.

You should now be able to reboot and see all required services start up in the correct order.

And done, that’s where I am ending this blog post,

1. we have setup n2n with supernodes and edge
2. generated valid sysvinit scripts

expect future posts to cover more advanced n2n configuration as I discover the options available.

Why you may ask? Performance!

To facilitate running wordpress on the smalest possible CloudServer I am using Varnish which using Apache as the backend, now with static files I can get all the blogging functionality without the need for Wordpress nor varnish, yet still uncached content can lead to increased load on the server.

Also wordpress does not lend itself to scalability especially with the at the time of writing schema and sql queries (percona-query-advisor flags up a few wordpress core sql queries as non scalable).

But this does not mean you need remove the option of using Wordpress, for you client for instance keeping wordpress in place can aid in content generation simply through ease of use.

1. wordpress used as normal
2. wordpress content is ported to markdown
3. markdown used to generate html

Now this does have caveats:

1. you’re going to need to maintain designs in wordpress templates and markdown (_layouts).
2. you’re going to need to handel any shortcode plugins in your export process.
3. you’re going to need to handel any other content modifying plugins in your export process.
4. any UGC / Dynamic requests will still need PHP.

But it essentially replaces the whole caching layer with static content which then can be pushed to CDN.

And with CDN’s now supporting index files (S3, and Coming Soon @ CloudFiles) in essence entire sites can be placed on CDN whilst maintaining ease of content generation.

Now don’t get me wrong, this requires a whole lot of “glue” to get working, but the potential for serving an entire web app from CDN without Origin pull / cache headers etc, saves a lot of systems time in scaling and adressing performance issues, or rather makes them “less critical” as the “business” part of the webapp is all on CDN.

I’m still getting to grips with Jekyll and by extention Octopress to see what it can achieve, so expect more posts.

This is a short guide on getting DRBD working @ Rackspace CloudServers Please note I will be resizing the root partition on the cloud server, this is not for the squeemish!

I highly recomend you only attempt this on new builds or throw away virtual machines, this only works for ext3 filesystems I have not tested nor do I know the implications of trying this on ext4!

rescue mode

In this first step we need to place the cloudserver in Rescue mode, so.

1. log into your rackspacecloud portal.
2. Hosting
3. Cloud Servers
4. Click server
5. Click Rescue

Read the prompt::

Placing your Cloud Server into rescue mode will allow you to debug system issues that are preventing it from booting to a usable state.

The rescue boot will use your current server IP and OS distro. Your original server devices will be accessible within the rescue mode as /dev/sda1 (root) and /dev/sda2 (swap). A temporary root password will be flashed when the image has booted. Note: the SSH server key will be different on the rescue image than your server.

Rescue mode is limited to 90 minutes, after which the rescue image is destroyed and the server will attempt to reboot. You may end the rescue mode at any time.

Note: In rescue mode /dev/xvda becomes /dev/sda

You can shose at this point to connect via SSH or via the Web Console, I will be opting for SSH.

Note: the host key is generated at startup, your ssh client will alert about a miss matching host key if you have allready connected to this host before. Or as a one off::

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no user@host

Do not use this reguarly, unless you enjoy the prospect of being an easy m.i.t.m target.

Now we need to record some information, however the text above is missleading /dev/sda does not exist as a device, this has been confirmed by Rackspace as information from the previous infrastructure (Slicehost I’d assume), the actual device is: /dev/xvdb (1 Data, 2 Swap) (And with any luck they will correct this soon within the system)

Now we need to do some information collecting so for now just mount /dev/xvdb1 onto /media::

mount /dev/xvdb1 /media
root@RESCUE ~]# df -h && df && df -B 4k && fdisk -l && fdisk -s /dev/xvdb1

Filesystem Size Used Avail Use% Mounted on /dev/xvda1 9.4G 894M 8.1G 10% / tmpfs 120M 0 120M 0% /dev/shm /dev/xvdb1 75G 2.1G 69G 3% /media Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 9804120 914776 8391324 10% / tmpfs 122228 0 122228 0% /dev/shm /dev/xvdb1 78440392 2157668 72298188 3% /media Filesystem 4K-blocks Used Available Use% Mounted on /dev/xvda1 2451030 228694 2097831 10% / tmpfs 30557 0 30557 0% /dev/shm /dev/xvdb1 19610098 539417 18074547 3% /media

Disk /dev/xvdb: 81.6 GB, 81604378624 bytes 255 heads, 63 sectors/track, 9921 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System

/dev/xvdb1 * 1 9922 79690752 83 Linux

Disk /dev/xvda: 10.2 GB, 10200547328 bytes 255 heads, 63 sectors/track, 1240 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0005781c

Device Boot      Start         End      Blocks   Id  System

/dev/xvda1 * 1 1241 9960448 83 Linux

Disk /dev/xvdd: 536 MB, 536870912 bytes 255 heads, 63 sectors/track, 65 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System

/dev/xvdd1 1 65 522112 82 Linux swap / Solaris

Disk /dev/xvdc: 536 MB, 536870912 bytes 255 heads, 63 sectors/track, 65 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System

/dev/xvdc1 1 65 522112 82 Linux swap / Solaris 79690752

Note: You could in get allthis information before dropping into rescue mode if you really wanted to.

Perparation

First a fsck we want it to report a clean FS without attempting any changes so we run this with the -n option, go grab a coffee this will take a while::

umount /media
fsck -n /dev/xvdb1 
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/: clean, 62421/9961472 files, 852007/19922688 blocks

Great we have a clean FS (note it will rpoert erros if the fs is still mounted!) we need to remove the journal, which essentially will turn a ext3 FS into ext2::

tune2fs -O ^has_journal /dev/xvdb1
tune2fs 1.41.12 (17-May-2010)

Force a filesystem check with e2fsck -f::

e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/: 62421/9961472 files (0.6% non-contiguous), 819205/19922688 blocks

Now we can resize the filesystem from the recon above we know that we are presentl using 2.1GB / 75GB so in this case I will “shave” 20GB off the top to become the DRBD volume::

resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/xvdb1 to 14080000 (4k) blocks.
The filesystem on /dev/xvdb1 is now 14080000 blocks long. 

Now we need to delete and recreate our partitions::

fdisk /dev/xvdb

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

     switch off the mode (command 'c') and change display units to
     sectors (command 'u').

Command (m for help): p

Disk /dev/xvdb: 81.6 GB, 81604378624 bytes 255 heads, 63 sectors/track, 9921 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System

/dev/xvdb1 * 1 9922 79690752 83 Linux

Command (m for help): d
Selected partition 1

Partition number (1-4): 1 First cylinder (1-9921, default 1): 1 Last cylinder, +cylinders or +size{K,M,G} (1-9921, default 9921): +59136000K

Command (m for help): a Partition number (1-4): 1

Command (m for help): w The partition table has been altered!

Calling ioctl() to re-read partition table. Syncing disks.

Wait how did I get 59136000K for the last cylinder? well we take 14080000 from resize2fs multiply by 4 and again by 1.05, why?

1. 4K blocksize
2. 1.05 gives us a 5% saftey buffer
3. final figure 59136000K
4. the option a is used to set the bootable flag back onto parition 1

Now we check the filesystem::

fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/xvdb1

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

Awww crap … right lets find out where that superblock is we run mke2fs with the -n flag this is a dry run, it will not write filesystem changes::

 mke2fs -n /dev/xvdb1 

mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 3702784 inodes, 14785816 blocks 739290 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 452 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
4096000, 7962624, 11239424

Ok NONE of these superblock worked for me … BLAM trashed VM, see why I suggested a throw away!

Credits

The following are information sources I used in the production of this method.

1. http://rackerhacker.com/2011/02/13/dual-primary-drbd-with-ocfs2/
2. http://www.howtoforge.com/linux_resizing_ext3_partitions i
3. http://www.cyberciti.biz/faq/recover-bad-superblock-from-corrupted-partition/

Well I for one say if that is the case then no one should be a “DevOps” without a background in Systems administration … let me explain.

Primarily I work with redhat rpm based systems for web application hosting at what I’d call an advanced level stracing, calling on linux c api’s as needed, fixing packaged and upstreaming the fixes, bug reporting etc (In my opinion something anyone using Opensource in their business should be doing!), I’m not going to go into complete detail on the tools and how I use them on a day to day basis as this moves from the point of this post entirely (that and it would take FAR too long to write …)

I also as part of my job I work in python, ruby, php, bash, tcl, c, c++, whatever tool is needed to do the job, let me say that again for clarity whatever tool is needed to do the job.

I could be a DBA, Sysadmin, TechSupport, Pentester at any given point of the day.

I analyse and profile web applications then go on to design hosting solutions for said applications.

I promote the use of SCM (Git in particular), unit testing and I’ve begun looking at Continuous integration methodologies.

I’m a commiter on the EPEL Openstack packages (Admittedly not as often as I would like at the moment … deadlines …), I also have upstream commits for libcloud and boxgrinder.

I work to the ethos that downtime is not acceptable, EVER! And if that means I have to profile, bugfix and code to ensure that is not the case then I will, I call it adapting and not being rigid.

I am presently looking at Chef to compliment my planned deploy of Openstack, for which I will be writing the configurations, this will in turn allow the development team to get on with their jobs, I already use kickstarts for my KVM deployments, Chef seems like the next logical step.

And whilst “The Cloud” has met with my skepticism, this is more to do with the over marketing claiming it is the solution to all your aliments … once you get past all the marketing fluff it is the way forward, and has been as such since a long time before “The Cloud” fluff came along.

So in short, I’m a Systems Administrator and I work damned hard to ensure those systems I administer stay online, if that means I need to work as a Developer, Pentester etc … then I will.

Whilst I can see that Devops in its current form could be stand alone from Systems Administration, it shouldn’t be …

You should not carry out Devops without a knowing the platforms you are deploying to, it’s like being a Cardiologist having spent 20 minutes on Operation (Yes overly melodramatic metaphor, remember uptime for me is that important.)

So what does that make me? aside from an overly paranoid uptime chanting nutter?

On Another note Saiweb.co.uk is 7 years old 26/03/2012 … I should really add more content …

What I did find a little lacking on the documentation side was the SCM integration (read: Source Control Management), git/svn etc …

In short so long as your rpm spec file is in your SCM (and it should be), moc will build your rpm from your sources in scm, which can be used for.

1. bleeding edge builds for testing
2. builds from “stable tags”

Yes yes yes … obvious I know …

So with no futher ado here is the syntax:

mock -r your_target --scm-enable --scm-option method=git --scm-option package=git_project --scm-option git_get='git clone git@git_ip_address:SCM_PKG.git SCM_PKG' --scm-option spec='SCM_PKG.spec' --scm-option branch=1-2 --scm-option write_tar=True -v

1. scm-enable - turns on the use of scm
2. scm-option - set an option for the scm in use

The above worked for me, you will need to adjust it acordingly, i.e. if your spec file is not named identically to that of your git project: –scm-option spec=’specfile_name.spec’

This will tie me over untill I get chance to play with my monkey farm

Firs you must meet the following conditions:

1. You are running gluster >= 3.0 <= 3.2 (May also work on 2.x I have not tested, and will not work with future versions if gluster change their use of xattrs)
2. You are running a replicated volume (Again I have not tested distributed volumes, in theory remove, re-add and rebalance will fix these)
3. You have a “good” copy of you data (This is essential this assume you have at least 1 brick with a good copy of the file system

Restrain and restore the “bad” brick

1. Shutdown all services that are using the mounted filesystem (i.e. httpd / nginx / *ftpd)
2. Unmount all the file systems on the node (glusterfs / nfs / etc …)
3. Grab a copy of stripxattr.py make sure you READ the README for installation requirements and usage
4. Run stripxattr.py against the backing filesystem on the “bad” node ONLY NOT AGAINST A GLUSTER MOUNT
5. From the “good” node, not rsync the data: rsync -gioprtv –progress /path/to/filesystem root@:/path/to
6. From the “good” node, trigger an ”auto heal” this will re-populate the xattr data (this must be done on a glusterfs mount not nfs/cifs/etc…)
7. Download listxattr.py once the self heal has completed see the README file for a “quick and dirty” consistency check
8. All being well you have now resolved a split-brain and can return your node to service

Current known gluster issues

1. NFS is much (48x in tests) faster for small files i.e. php webapps, but does not support distributed locking meaning: all nodes can write to the same file at the same time, this is what cause our original split brain

So what is the resolution int his case?

Selective use, use glusterfs for filesystems that you need distributed locking, often in large production deploys php files will not change often, in this case NFS is perfect.

If you are still writing php sessions to a file system then STOP IT and use a database! (Better yet use memcache).

There’s been a lot developing over the last few months, Openstack being one of my main focuses along with overhauling and provision new internal systems for Openstack to run upon, I have a plan so to speak …

I have some Openstack posts coming I just need to ensure that all parties are happy with me posting the information “in the clear” so to speak.

In one of the worst customer service experiences I’ve ever had the miss fortune to have been a part of all access was closed to our in office version control systems due to “high usage”, now this is a pretty essential service as you might imagine, how then to restore access, when the restrictions are beyond your control? (And I mean EVERY inbound port was dead …)

Fortunately it would seem outbound SSH was not affected, so after much vocal drawing and re-drawing many times over on the whiteboard I had a cunning plan …

Using 3 linux devices I would create the following.

1. A device through which using host entries / dns changes the version control would be available whilst not actually running on the box itself.

2. An in house device which would be the device on which the tunnels are created in the first place.

3. The device(s) on which the version control systems reside.

Gateway device

On the gateway device sshd_config needs to be updated with:

GatewayPorts yes

And sshd reloaded.

Also if you are using a local firewall (i.e. iptables) you will need to setup the appropriate rules as if the service were running natively on the device

Pivot Device

Generate rsa ssh keys and deploy your id_rsa.pub to the gateway device, (update sshd_config to enable RSA Auth if required)

The tunnel.

ssh <Gateway Device> -l root -g -N -R 0.0.0.0:<Service Port>:10.0.0.1:<Service Port>  -vvv

Now you only really need to use root if the port you need to gateway is a privileged port (<1024).

Here 10.0.0.1 is the internal address of the device the connection should “pivot” onto.

Once the tunnel was in place the services could be reached via the gateway device, this was essentially a “poor mans” NAT in a time of need, I really do not suggest this for long term use.

for i in {60..200}; do ping -c 1 -W 1 192.168.1.$i > /dev/null; ([[ $? == 0 ]] && echo "$i UP" || echo "$i DOWN");  done
1 UP
2 DOWN
3 UP
...

Note: for OSX use “ping -c 1 -t 1”

Chaining “UP” hosts for a quick (syn) port scan

for i in {60..200}; do ping -c 1 -W 1 192.168.1.$i > /dev/null; ([[ $? == 0 ]] && nc -v -n -z -w1 192.168.1.$i 20-22); done
(UNKNOWN) [192.168.1.1] 22 (ssh) open
(UNKNOWN) [192.168.1.3] 22 (ssh) open

Recover from a bad mysql password set (Update mysql.users set password=’Iforgotawherestatemenlulz’)

Assumes for every user there is an @localhost host, grabs the in memory password hash and resets

mysql -Bse 'Select distinct(user) from mysql.user;' | while read uname; do mysql -Bse "show grants for '$uname'@'localhost';" 2>&1 | grep IDENTIFIED | grep -v 'root' | grep -v 'ERROR' | sed 's|GRANT USAGE ON *.* TO ||g' | sed "s|@'localhost' IDENTIFIED BY PASSWORD||g" | awk '{print "Update user set Password="$2" where User="$1";"}' | mysql mysql; done

If you’ve run FLUSH PRIVILEGES; however you == b0ned.

Quick substitute and run

Command1:

ping -c 1 -t 1 192.168.1.1

Opps that’s OSX synatx

Command2:

^-t 1^-W 1

et voila corrected syntax.

Shortcuts

!! - Execute last command !ping - Execute last ping command, can be used to !any command just be careful. ctrl+r - reverse search, just start typing the cmd for it to search your history, hit tab to complete ctrl+a - jump to beginning of line ctrl+e - jump to end of the line

cURL FU

curl -I -L blahblah.tld - Run a HEAD and follow redirects (very handy for quicklooking @ bit.ly short URLS before hitting them in a browser).

python FU

python -m SimpleHTTPServer - serves the current pwd as a browseable directory (Very cool but VERY insecure) python -m cProfile script.py - generate trace stats for a script execution (Very handy for finding excessive loops)

DNS Fu

Wikipedia over DNS:

host -t txt fu.wp.dg.cx

fu.wp.dg.cx descriptive text “Fu may refer to: Fu (Technology, especially computer related) (used as a suffix) - relating to a person - Possessing superior skills in an art\; relating to an artifact - representing an expression of high art. code-fu, Perl-fu, C-fu, etc, Fu (literature),” ” a Chinese genre of rhymed prose, Fu (kana), a symbol in Japanese syllabaries, Fu County, in Shaanxi, China, Fu Foundation… http://a.vu/w:Fu”

Useful on some public wifi connections if you just want to look something up quick (dns is not always re-written).

Get all MX servers for a domain:

dig google.co.uk MX

; <<>> DiG 9.6.0-APPLE-P2 <<>> google.co.uk MX ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64165 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION: ;google.co.uk. IN MX

;; ANSWER SECTION: google.co.uk. 10800 IN MX 10 google.com.s9a1.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9a2.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9b1.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9b2.psmtp.com.

;; AUTHORITY SECTION: google.co.uk. 59925 IN NS ns2.google.com. google.co.uk. 59925 IN NS ns3.google.com. google.co.uk. 59925 IN NS ns4.google.com. google.co.uk. 59925 IN NS ns1.google.com.

;; ADDITIONAL SECTION: ns1.google.com. 158334 IN A 216.239.32.10 ns2.google.com. 158334 IN A 216.239.34.10 ns3.google.com. 158741 IN A 216.239.36.10 ns4.google.com. 158334 IN A 216.239.38.10

;; Query time: 68 msec ;; SERVER: ;; WHEN: Mon Sep 26 16:41:26 2011 ;; MSG SIZE rcvd: 310

mySQL FU

in one line, take a database, in stream replace content and stream into another db.

mysqldump original_db | sed ‘s/content_or_regex_to_replace/content_or_backref_replacement/g’ | mysql destination_db

Credit goes to d3in0s for his awesome forum post.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport
Usage: airport <interface> <verb> <options>

    <interface>
    If an interface is not specified, airport will use the first AirPort interface on the system.

    <verb is one of the following:
    prefs   If specified with no key value pairs, displays a subset of AirPort preferences for
        the specified interface.

        Preferences may be configured using key=value syntax. Keys and possible values are specified below.
        Boolean settings may be configured using 'YES' and 'NO'.

        DisconnectOnLogout (Boolean)
        JoinMode (String)
            Automatic
            Preferred
            Ranked
            Recent
            Strongest
        JoinModeFallback (String)
            Prompt
            JoinOpen
            KeepLooking
            DoNothing
        RememberRecentNetworks (Boolean)
        RequireAdmin (Boolean)
        RequireAdminIBSS (Boolean)
        RequireAdminNetworkChange (Boolean)
        RequireAdminPowerToggle (Boolean)
        WoWEnabled (Boolean)

    logger  Monitor the driver's logging facility.

   sniff   If a channel number is specified, airportd will attempt to configure the interface
       to use that channel before it begins sniffing 802.11 frames. Captures files are saved to /tmp.
       Requires super user privileges.

   debug   Enable debug logging. A debug log setting may be enabled by prefixing it with a '+', and disabled
       by prefixing it with a '-'.

        AirPort Userland Debug Flags
            DriverDiscovery
            DriverEvent
            Info
            SystemConfiguration
            UserEvent
            PreferredNetworks
            AutoJoin
            IPC
            Scan
            802.1x
            Assoc
            Keychain
            RSNAuth
            WoW
            AllUserland - Enable/Disable all userland debug flags

        AirPort Driver Common Flags
            DriverInfo
            DriverError
            DriverWPA
            DriverScan
            AllDriver - Enable/Disable all driver debug flags

        AirPort Driver Vendor Flags
            VendorAssoc
            VendorConnection
            AllVendor - Enable/Disable all vendor debug flags

        AirPort Global Flags
            LogFile - Save all AirPort logs to /var/log/airport.log

<options> is one of the following:
    No options currently defined.

Examples:

Configuring preferences (requires admin privileges)
    sudo airport en1 prefs JoinMode=Preferred RememberRecentNetworks=NO RequireAdmin=YES

Sniffing on channel 1:
    airport en1 sniff 1


LEGACY COMMANDS:
Supported arguments:
 -c[<arg>] --channel=[<arg>]    Set arbitrary channel on the card
 -z        --disassociate       Disassociate from any network
 -I        --getinfo            Print current wireless status, e.g. signal info, BSSID, port type etc.
 -s[<arg>] --scan=[<arg>]       Perform a wireless broadcast scan.
                   Will perform a directed scan if the optional <arg> is provided
 -x        --xml                Print info as XML
 -P        --psk                Create PSK from specified pass phrase and SSID.
                   The following additional arguments must be specified with this command:
                                  --password=<arg>  Specify a WPA password
                                  --ssid=<arg>      Specify SSID when creating a PSK
 -h        --help               Show this help

Credit goes to d3in0s post showing true forum awesomeness.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I
     agrCtlRSSI: -40
     agrExtRSSI: 0
    agrCtlNoise: -92
    agrExtNoise: 0
          state: running
        op mode: station 
     lastTxRate: 54
        maxRate: 54
lastAssocStatus: 0
    802.11 auth: open
      link auth: wpa2-psk
          BSSID: <removed>
           SSID: <removed>
            MCS: -1
        channel: 6
/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -s
                            SSID BSSID             RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
                          <removed> <removed> -41  6       N  -- WPA(PSK/AES,TKIP/TKIP) WPA2(PSK/AES,TKIP/TKIP)

Doing a frame cap.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport en1 sniff 6
Capturing 802.11 frames on en1.

You will see your airport icon changes to now hit ctrl+c to stop the cap

^CSession saved to /tmp/airportSniff813ZrA.cap.

curl -O http://python-distribute.org/distribute_setup.py
python3 distribute_setup.py

Enjoy!

Out of the box, php will use sendmail, and it will do so as follows.

1. mail() forks sendmail process
2. sendmail attempts to send email to destination server
3. sendmail returns on send complete

sendmail_path = /usr/sbin/sendmail -t -i -O DeliveryMode=b

This sets the delivery mode to background, sendmail will return to php near instantly and send the email in the background by placing in into a queue.

TL;DR

Put the above in your php.ini to not hang around to sendmail, and hav it return instantly.

First off we need to take a hash dump:

mysql -Bse 'select distinct(password),user from mysql.user;' > hashdump.txt

Now of course you can do the same using SQL Injection etc (WHY when you have SQLi already? duh privilege escalation!) , I’m going to cover this from the perspective that you are the administrator looking to strengthen your security …

Now you have your hashdump you need a hash table with the equivelent passwords within it, for this you will need 2 things

1. A dictionary file
2. https://github.com/Oneiroi/PenTesting/blob/master/crypto/generators/mysql/csv_gen.py

The python script above I wrote to use multiprocessing to map words onto the hash function, and I have had it grind through mySQL hashes at a rate of ~98k per second, there is no “lookup” script at this time though one is currently being written.

./csv_gen.py -f /path/to/wordlist.txt -o /output/path/to/output.csv -t <max threads, default 1> [-l optional use legacy hash]

once this has ground through your wordlist you will have a CSV file, which will be in the format , the script defaults to the new PASSWORD() function, if you are using old_password=1 in your configuration then pass the -l flag to use legacy hashing instead.

ok let’s assume the following fictional scenario

1. old_passwords is in use, and we want chip’s password
2. 077b91e3491e2fdd chip

grep 077b91e3491e2fdd output.txt
077b91e3491e2fdd,a

1. Chip has a password that is just he letter “a” which he will tell you is the best password ever …

And that’s about a simple as it gets you generate a set of hashes and you compare known hashes to your generate set to see if you can discern simple passwords, hopefully going on then to chastise the user and instructing them on proper password etiquette, there are more complicated methods of getting the password from the hash, in the case of old_passwords I believe it is possible to reverse the hash to get the original string for one (so don’t use old_passwords!)

If you go on to use my python scripts, please let me know how they perform, my test were carried out using an intel i5, I’d love to know how they perform on other CPUs.

In this post I will cover the basic generation of a LAMP (Linux Apache MySQL PHP) stack CentOS appliance, nothing to complicated I assure you, and no magic like auto deployment spin up etc … that’s for later … no skipping ahead!

First of all you’re going to need boxgrinder I recommend downloading the Meta appliance, as it has all the tools you need already.

Now I am covering the following.

1. basic use of boxgrinder-build on the meta appliance
2. creation of centos lampstack basic
3. deploying the image to KVM

I’m going to have to assume that you are capable of downloading and starting up the meta appliance yourself, and focus more on the stack setup.

Grinding your VM

Ok so you are going to need a YAML file defining the CentOS lamp stack, save this on your meta appliance as CentOS-lamp.yaml

name: CentOS-lamp
summary: Generic CentOS 5.6 LAMP stack, with some apache & php tuning
version: 1
release: 0
hardware:
cpus: 2
memory: 1024
partitions:
"/":
size: 5
"/var/www":
size: 15
os:
name: centos
version: 5
password: changeme

On your Meta appliance run.

boxgrinder-build -d CentOS-lamp.appl

This process will take a while, so go and get a coffee, this will produce ./build/appliances/x86_64/centos/5/CentOS-lamp/CentOS-lamp-sda.raw once complete, if you run into issues the -d flag is “debug” paste your log output int the comments and I will do my best to diagnose and fix your issue.

Deploying to KVM

boxgrinder has SFTP support for pushing to remote servers, you can use this if you like to automate the “push” of the image to your KVM server, at the moment automated deployment to KVM is not support but may be coming soon.

Assuming you have placed you image in /var/lib/libvirt/images/

virt-install -n "Saiweb - CentOS-lamp Demo" -r 1024 --arch=x86_64 --vcpus=1 --os-type=linux --os-variant=rhel5.4 --disk path=/var/lib/libvirt/images/CentOS-lamp.raw,size=20,cache=none,device=disk --accelerate --network=bridge:br0 --vnc --import

Post startup

this is a VERY basic setup I have not covered any of the post install options in this post (but I will in future posts), so.

chkconfig httpd on && service httpd start
chkconfig mysqld on && service mysqld start

This will set your services to automatically start at startup, and start them.

In short replace your usual

sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged."; #uses error function to return simple confirmation
        }
}
sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache."; #request to purge none existant item
        }
}

with

sub vcl_recv {
        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                ban("req.url ~ "+req.url+" && req.http.host == "+req.http.host);
                error 200 "Purged.";
        }
...

Substituting “~ purge” with your ACL name, the above implement wild card purging aswell, if you do not want this and only want PURGE for the exact passed URL replace

“req.url ~ “+req.url

with

“req.url == “+req.url

Using the same tests as

ab -c 100 -n 500 -g ./saiweb-nocache-nogzip.bpl http://www.saiweb.co.uk/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.saiweb.co.uk (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Finished 500 requests

Server Software: Apache Server Hostname: www.saiweb.co.uk Server Port: 80

Document Path: / Document Length: 92719 bytes

Concurrency Level: 100 Time taken for tests: 0.184 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 47597095 bytes HTML transferred: 47379409 bytes Requests per second: 2716.92 [#/sec] (mean) Time per request: 36.806 [ms] (mean) Time per request: 0.368 [ms] (mean, across all concurrent requests) Transfer rate: 252573.13 [Kbytes/sec] received

Connection Times (ms)

          min  mean[+/-sd] median   max

Connect: 1 4 1.1 4 6 Processing: 9 31 7.0 32 47 Waiting: 2 7 5.7 4 26 Total: 15 35 6.8 36 53

Percentage of the requests served within a certain time (ms) 50% 36 66% 38 75% 39 80% 39 90% 41 95% 44 98% 48 99% 51 100% 53 (longest request)

2716.92 requests per second with a server load average of 0.1, and in this case varnish is serving cache from disk.

Caching using varnish (Or even nginx / mod_cache) means that PHP does not get executed at all, the cache system grabs the cache content and serves it.

This of course has the benefit of reducing the CPU and memory resources needed for the running of your application, but it does have some caveats.

• This only works for GET requests, and content not reliant on Cookies (Truely dynamic content will not cache)
• But on the “flipside” Varnish supports ESI, which when setup correctly you can target the dynamic sections of a pag for “passthrough” and have the rest cached
More details to come, as I have time to add them I have have a lot of posts to make on boxgrinder, KVM, libvirtd etc. ]]> 2011-07-21T13:58:01+01:00 http://saiweb.co.uk/mac/fixing-osx-lion-afp-the-version-of-the-server-you-are-trying-to-connect-to-is-not-supported For those using netatalk for AFP shares in this case I am using CentOS, the EL5 compiles are missing the configure lines for the dhx2 extension, which is required by OSX Lion, if you are running x86_64 you can grab this file: netatalk-2.0.5-2.x86_64.rpm I have also emailed the Package maintainer @ EPEL with the changes I have made for this RPM so I would like to think that -2 will be available from EPEL soon.

Let me know if you have any issues with my RPM.

UPDATE: Official Rebuild in testing

]]>