rancid and relays using usercmd patch

As I've blogged before, rancid is a really awesome new cisco config differ.

I monitor about 100 devices, and all was well so far.
Now I've come across a device that I can't reach directly from my rancid host. It's not possible to create a direct network connection, or to use ssh tunnels or netcat to forward those ports using a relay host.
Instead I found a much simpler and more elegant solution. Albeit this involves patching rancid.


After playing with ProxyCommands and netcat I came across this patch on the rancid-discuss list from 2006.
The principle seems simple, elegant but still straight forward. Extending the clogin script to define a new method that can use custom commands to establish the connection.
This is accomplished by adding two new commands.

add method testrouter {usercmd}
add usercmd testrouter {ssh} {-t} {termserver01} {cu -l /dev/tty01 -s 9600}
add usercmd_chat testrouter {Connected.} {\r}

In my case I wanted to ssh to the relay host, and then telnet to my device from there. This is how I set that up:

add method my-alias usercmd
add usercmd my-alias {ssh} {-t} {rancid@my-relay-host} {telnet my-device}
add user my-alias rancid
add password my-alias rancid

Two caveats to this situation:

• The username on my-relay-host must match the username used on the device my-alias.
• This requires password less authentication to my-relay-host. I've used ssh standard key exchange.

Read More

TortoiseSVN 1.8.0 and NTLM Authentication on Windows

Subversion 1.8.0 was released recently, and my favorite svn client was updated, too. I had hoped this update to go down smooth, just like going from 1.6 to 1.7 in the past.
This time around there are some authentication issues it seems. TortoiseSVN has dropped neon and relies now solely on serf to handle HTTP(S) authentication.

The release notes mention some server-side configuration changes for optimal performance, but they fail to mention that serf doesn't play nice with NTLM authentication. It does work well with Kerberos though.

So, if you've set up Apache like I described here or here, just add this line to make Kerberos the default and have  TSVN 1.8.0 play nice with single-sign-on.

SSPIPackage Negotiate

This will change the default authentication schema (NTLM) to Kerberos. This will cause TortoiseSVN < 1.8 to prompt for username and password, because the underlying neon library cannot handle Kerberos. TortoiseSVN 1.8 and later will be fine though.
You can force older clients to use serf for specific servers though. I've previously written about it here.

There is a big discussion going on in the subversion-dev mailing list about how to best fix this behavior. It is expected that TortoiseSVN 1.8.1 will include a fix for this issue.

This issue and the fact that most SVN tools still haven't been built on top of subversion 1.8 means that you should stay on TortoiseSVN 1.7.x for now.

Read More

CollabNet Subversion Edge 4.0 with SSPI WSGI and trac

CollabNet released Subversion Edge 4.0 recently, as they've written here. As you know I've been using this to host our Subversion and trac server. I've made a few tweaks and additions but I've never had any issues with updating Subversion Edge. Until now, that is.


This time Collabnet decided to update Apache from 2.2 to 2.4. This breaks compatibility with some of the modules I use. Let's look at mod_auth_sspi first.

The module mod_auth_sspi is used for authentication against an Active Directory. When configured properly, it allows clients to seamlessly connect using single-sign-on. In a corporate world this is a must-have. Unfortunately this module is no longer supported on Apache-2.4.
There is a replacement, however: mod_authnz_sspi. It can almost act as a drop-in replacement for mod_auth_sspi. No wonder, since it started out as a fork.
The changes are small. Instead of

require DOMAIN\Group

the correct syntax is now

require sspi-group  DOMAIN\Group

The module name change too, so the module has to be loaded like this:

LoadModule authnz_sspi_module lib/modules/mod_authnz_sspi.so

Don't worry, I'll post a complete apache config at the end.

The other module that acted up was mod_wsgi. Now if you don't use trac on Subversion Edge you might not even notice this. Even if you use trac, you can still make it work using CGI on windows. I'd really discourage you from doing that, since it's really abnormally slow.
A google search reveals a couple of mod_wsgi binaries built for Apache-2.4. None of them work with Subversion Edge 4.0 (at least none that I've found).

So I had to build it myself. The requirements are:

• MS Visual C compiler (Visual Studio Express will do)
• MS Windows SDK
• Apache 2.4 development/header files
• Python-2.7.1 development/header files

I downloaded the 3.4 source here. I got Apache 2.4.4 here.I got Python 2.7.1 here, because that's the version that's bundled with Collabnet Subversion Edge.
Note the folders you installed/extracted those files, you'll need them soon. There are a few makefiles for various Apache and Python versions, but none for Apache 2.4 and Python 2.7. So I had to create my own. Here are the contents:

CPPFLAGS = \
 /DWIN32 \
 /DNDEBUG \
 /I"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include" \
 /I"C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1\Include" \
 /I"C:\Apache24\include" \
 /I"C:\Python27\include"

CFLAGS = \
 /MD \
 /GF \
 /Gy \
 /O2 \
 /Wall \
 /Zc:wchar_t \
 /Zc:forScope

LDFLAGS = \
 /link \
 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib" \
 "/LIBPATH:C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1\Lib" \
 "/LIBPATH:C:\Apache24\lib" \
 "/LIBPATH:C:\Python27\libs" \
 /OPT:REF \
 /OPT:ICF=2 \
 /RELEASE \
 /SUBSYSTEM:WINDOWS \
 /MANIFEST

LDLIBS = \
 python27.lib \
 libhttpd.lib \
 libapr-1.lib \
 libaprutil-1.lib

mod_wsgi.so : mod_wsgi.c
    cl $(CPPFLAGS) $(CFLAGS) $? /LD $(LDFLAGS) $(LDLIBS) /OUT:$@
    mt -manifest $@.manifest -outputresource:$@;2

clean :
    del *.obj *.so *.so.manifest *.lib *.exp

As you can see I've installed Python to C:\Python27 and Apache to C:\Apache24. Feel free to adjust your paths. To actually compile mod_wsgi you have to open a Visual Studio Command Prompt, otherwise some environment variables will be missing and the compiler fails.
I got a lot (100+) warnings from the compiler, but in the end it worked:

Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:mod_wsgi.dll
/dll
/implib:mod_wsgi.lib
"/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib"
"/LIBPATH:C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1\Lib"
/LIBPATH:C:\temp\Apache24\lib
/LIBPATH:C:\Python27-x86\libs
/OPT:REF
/OPT:ICF=2
/RELEASE
/SUBSYSTEM:WINDOWS
/MANIFEST
python27.lib
libhttpd.lib
libapr-1.lib
libaprutil-1.lib
/OUT:mod_wsgi.so
mod_wsgi.obj
   Creating library mod_wsgi.lib and object mod_wsgi.exp
        mt -manifest mod_wsgi.so.manifest -outputresource:mod_wsgi.so;2
Microsoft (R) Manifest Tool version 5.2.3790.2076
Copyright (c) Microsoft Corporation 2005.
All rights reserved.

I copied to mod_wsgi.so file to the csvn/lib/modules/ folder on my Collabnet Subversion Edge server. With all modules in place, it's time to tweak the apache config files.

This is my httpd.conf as it currently resides in csvn/data/conf. The Subversion Edge web interface alters and changes the file, so editing it is always risky. This version seems stable enough though.

ServerRoot "D:\csvn"
Include "data/conf/csvn_modules_httpd.conf"
Include "data/conf/csvn_main_httpd.conf"
Include "data/conf/csvn_logging.conf"
Include "data/conf/csvn_default_dirs_httpd.conf"
#Include "data/conf/svn_viewvc_httpd.conf"
Include "data/conf/csvn_misc_httpd.conf"

LoadModule authnz_sspi_module lib/modules/mod_authnz_sspi.so

##########################################

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht">
    Require all denied
</FilesMatch>

LimitXMLRequestBody 0
ServerSignature  Off
ServerTokens  Prod

TypesConfig "D:\csvn\data/conf/mime.types"
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

# commented out for safety - contets dumped below
# Include "D:\csvn\data/conf/svn_viewvc_httpd.conf"

Include "D:\csvn\data/conf/ctf_httpd.conf"

SSLEngine On

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/svn)$ $1/
<Location /svn/>  
    DAV svn
    SVNParentPath "D:\csvn\data\repositories"
    SVNReposName "CollabNet Subversion Repository"
#   AuthzSVNAccessFile "D:\csvn\data/conf/svn_access_file"
    SVNListParentPath On

    AuthType SSPI
    SSPIAuth On
    SSPIAuthoritative On
    SSPIDomain DOMAIN
    SSPIOfferBasic On
    SSPIOmitDomain On
    <RequireAll>
        <RequireAny>
        require sspi-group DOMAIN\Group
        </RequireAny>
    </RequireAll>
</Location>

<Directory "D:\csvn\www\viewVC/docroot">
  AllowOverride None
  Options None
  Require all granted
</Directory>
Alias /viewvc-static "D:\csvn\www\viewVC/docroot"

<Location /viewvc-static>
  Require all granted
</Location>

ScriptAlias /viewvc "D:\csvn\bin/cgi-bin/viewvc.cgi"

<Location /viewvc>
    AddDefaultCharset UTF-8
    SetEnv CSVN_HOME "D:\csvn"

    AuthType SSPI
    SSPIAuth On
    SSPIAuthoritative On
    SSPIDomain DOMAIN
    SSPIOfferBasic On
    SSPIOmitDomain On
    require sspi-group DOMAIN\Group
</Location>

LoadModule wsgi_module lib/modules/mod_wsgi.so
WSGIScriptAlias /trac "data/conf/trac.wsgi"

<Location /trac>
    WSGIApplicationGroup %{GLOBAL}

    AuthType SSPI
    AuthName "Trac"
    SSPIAuth On
    SSPIAuthoritative On
    SSPIDomain DOMAIN
    SSPIOfferBasic On
    SSPIOmitDomain On
    <RequireAll>
        <RequireAny>
        require sspi-group DOMAIN\Group
        </RequireAny>
    </RequireAll>
</Location>

So basically the contents of the file svn_viewvc_httpd.conf were appended to the httpd.conf file, so I could adjust the Auth settings.
If you don't use trac, you can obviously skip WSGI and the trac location.

The last missing piece to have trac working is the trac.wsgi script. You can create have trac-admin create a template for you by running

trac-admin <env> deploy <dir>

I did that, and tweaked the resulting file a bit. I'm not sure whether that's the correct way of doing it, but this hack has been working fine on another server for years already.
Here is my version

import os

def application(environ, start_request):
    os.environ['TRAC_ENV_PARENT_DIR'] = 'd:\\trac'
#    if not 'trac.env_parent_dir' in environ:
#        environ.setdefault('trac.env_path', 'd:\\trac\\myproject')
    if 'PYTHON_EGG_CACHE' in environ:                                          
        os.environ['PYTHON_EGG_CACHE'] = environ['PYTHON_EGG_CACHE']
    elif 'trac.env_path' in environ:
        os.environ['PYTHON_EGG_CACHE'] = \
            os.path.join(environ['trac.env_path'], '.egg-cache')
    elif 'trac.env_parent_dir' in environ:
        os.environ['PYTHON_EGG_CACHE'] = \
            os.path.join(environ['trac.env_parent_dir'], '.egg-cache')
    from trac.web.main import dispatch_request
    return dispatch_request(environ, start_request)

Now when I browse to /trac on my Collabnet Subversion Edge server, I am presented the "Available Projects" overview.

Trac will work, but it requires python-subversion bindings. Luckily Collabnet supplies these with Subversion edge. More details here. They can be found in lib/svn-python. Just copy the contents of that folder to csvn/Python-2.5/lib/site-packages. Without doing that Trac will still work but lack svn support.

That's all folks!

If anyone is interested, I can make my build of mod_wsgi available.

Read More

XenServer 6.1, Dell OpenManage and SNMP

I finally updated our productive environment to XenServer 6.1. The upgrade went without a hitch, but there are still many issues with Windows VMs and the XenTools. In the end completely removing the XenTools and reinstalling them did the trick.

Anyway, this post is about Dell OpenManage and SNMP. I've found a spot-on post in the Citrix forums here - thanks Josh Phillips. Read on for the details.


Those are the exact steps:

1. Follow the steps in the Dell OM 7.2 Linux repository:
   wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
   yum install srvadmin-all
2. The latter command complanis about missing dependencies (e.g. libxslt.so.1 ) so let's enable the CentOS-Base repo to fix it. Edit /etc/yum.repos.d/CentOS-Base.repo and change [Base] to "enabled=1".
3. yum install srvadmin-allThis will now proceed.
4. Firewall adjustments - I'm always extra extra careful when messing with iptables rules. It's just too easy to make a typo and lock yourself out. I'm much more comfortable by making those adjustements in the system-config GUI.
   Note: The default putty console window size is too small to show the whole dialog, at least on my XenServers. So I had to resize it before running the command.
   Also if window borders look weird, make sure to change Putty Translation options to UTF-8.
   
1. Run "system-config-securitylevel-tui"
2. Go to "Customize"
3. Go to the text box "Other Ports" in "Allow incoming". Append the following strings. Please note the comma!
1. ,1311:tcp
   This is to allow access to the OMSA web interface
2. ,161:udp
   This will allow incoming SNMP requests
5. XenServer snmpd is quite restrictive by default. In our environment it's ok to respond to all requests, read-only. We do this by adding the line "view systemview included .1" after the line "view systemview included snmp" in /etc/snmp/snmpd.conf.
6. Now restart snmpd by running:
   service snmpd restart
7. And let's start Dell OMSA by running:
   /opt/dell/srvadmin/sbin/srvadmin-services.sh start

Make sure you're able to log in to the web interface on https://<server>:1311. Also your XenServer host should now respond to any SNMP queries.
The whole process takes less than 10 minutes, depending on your internet connection. About 200MBs need to be downloaded. The XenServer does not need to be rebooted!

Read More

OpenMediaVault & XBMC

I recently stumbled upon OpenMediaVault (OMV), a Linux distribution specific for NASes based on debian squeeze. It's inspired by FreeNas as I gathered.


There is a fast-growing community around OMV that offers guides and how-tos. I tried to give the VM a shot and wanted to see how far XBMC has come. There is a detailed guide of ow to install XBMC on OMV here and here. Unfortunately I hit a roadblock because I was unable to install Oracle Java 7. The command 

sudo add-apt-repository ppa:webupd8team/java

Was not recognized. After some digging I found out how to add the correct repo for debian squeeze on the webupd8 blog here. Here are the relevant commands:

su -
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
apt-get update
apt-get install oracle-java7-installer
exit

I was not able to get the keys added, so I had to install the package without verification, which worked fine too.

Update:
I had to update libtinyxml and libtinyxml-dev from the Debian wheezy repository to get xbmc to compile.
The two packages can be found here and here. 

Update 2:
I had to remove any and all taglib and depending packets and compiled and installed taglib manually from source.
Here's how:

wget https://github.com/downloads/taglib/tagl...1.8.tar.gz
tar xzf taglib-1.8.tar.gz
cd taglib-1.8
cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_RELEASE_TYPE=Release .
make
make install

Make would complain about some missing libraries. So I had to run make with additional options:

export CFLAGS="-ldl"
export CXXFLAGS="-ldl"

 Which stops xbmc from complaining about missing libraries.

Read More

SQL Server Update Woes

There are various issues that can happen when trying to update SQL Server. For almost all there are workarounds, and they usually work well.

Today I had a trickier experience, and thought I'd share my findings. I wanted to install KB2494113 - "Security Update for SQL Server 2005 Service Pack 3". I got error 57A through Windows Updates. So I had to resort to downloading and installing the update manually.



The installation failed, but at least the summary had the real error message:

**********************************************************************************
Product Installation Status
Product                   : SQL Server Database Services 2005 (SQLEXPRESS)
Product Version (Previous): 4053
Product Version (Final)   :
Status                    : Failure
Log File                  : C:\Program Files (x86)\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9_Hotfix_KB2494113_sqlrun_sql.msp.log
SQL Express Features      :
Error Number              : 1402
Error Description         : MSP Error: 1402  Could not open key: UNKNOWN\Components\GUID1\GUID2.   Verify that you have sufficient access to that key, or contact your support personnel.
----------------------------------------------------------------------------------

The relevant registry path can be found in 

HKLM\Software\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\GUID1\GUID2

Where Guid1 and Guid2 are the specific paths from the error log above. Sure enough, I was not able to even access these keys as Administrator.


This blog post had the solution, although one important step was left out in workaround 1. In order to change the permissions of those folders, you first have to claim ownership.
I didn't feel comfortable changing ownership of the whole Components key, so I just changed the keys that were referenced in the error log above, namely GUID1\GUID2.
It's bad enough having to change the registry as local system, I didn't want to further endanger the stability of the system.

That means going through the following process:

1. Attempt to install KB2494113
2. On failure, get the registry key from the error from "View Summary"
3. Run "psexec -i -s cmd.exe", then invoke "regedit"
4. Navigate to HKLM\Software\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\GUID1
5. Right click the key GUID1 and select "Permissions". Click "Advanced".
   
6. In the Permissions, Owner tab: Check "Replace owner on subcontainers and objects" then chose a valid user or group. Click "Apply". You'll get a message that tells you that it couldn't set the owner on some of the subkeys. That's alright in this case.
7. In the Permissions, Permissions tab check "Replace all child object permissions with inheritable permissions frmo this object" and click "OK". Answer "Yes" when prompted.
8. Goto 1 until the installation doesn't fail

In my case I had to tweak two dozen or so reg keys, but then it worked.

Read More

XenServer 6.1

As I've previously written about here, I'm using the Citrix XenServer. I've been using the free version in a production environment for a few years now. Sure, there were minor issues or hiccups, but no deal breakers.

I'm still running XenServer 5.6, although XenServer 6 and even 6.1 were already released.
Why didn't I upgrade yet?



Well, for one there is the mantra of "Never change a running system." Other than that, what's to be gained from an upgrade? For one the list of features in the free edition of XenServer has remained more or less constant. 
On the other hand due to changes in the XenTools in 6.1 a lot of problems were introduced for Windows VMs. There are a lot of problem reports on the citrix forums, so I decided to wait until those were addressed.

Read More