Using AutoitX as a DLL in Python

We’ve been using AutoIt for a while now at the shop.  We do a LOT of repetitive stuff; for instance every computer that we clean up gets pretty much the exact same process.  Any time you find yourself performing the same steps over and over again on a computer by hand, that’s a sign you should be figuring out a way to let the computer do it for you (that’s what they’re for!) and AutoIt is one of the easiest ways to automate GUI interaction for those pesky Windows programs that don’t allow automation via command line options.

For a while now though I’ve been struggling with the fact that while AutoIt is really good at GUI automation, it isn’t very good at things like manipulating complex configuration data: a job which, unsurprisingly, often goes hand-in-hand with automating a few mouse clicks.  Often you’ll use automated clicks to install a program, but find that directly editing a config file is an easier way to configure it and that’s when AutoIt falls short.  It has file manipulation tools but they are very basic.  After all, it’s a GUI automation kit NOT a full fledged programming language; I don’t blame the AutoIt folks one bit for doing one thing and doing it well.  That’s the mantra us ‘nix folks live by!  So on and off over time I’ve sort of peaked around for a different solution, one that gives me both solid GUI automation and a full-fledged programming language with lots of good modules/libraries for various tasks.

Other GUI automation tools were eliminated pretty quickly.  I didn’t find anything that was a solid or feature complete as AutoIt.  That left me with looking for a way to glue multiple tools together that did NOT result in a house-of-cards setup that would be nearly impossible to replicate in the case of a failure or that would rely on the perfect alignment of planets to run reliably.  It didn’t take me long to realize that AutoItX was my best bet.

AutoItX (available on the main AutoIt download page) is basically a library version of AutoIt that can be used from other programming languages via a DLL or the Windows COM system.  It comes with some documentation for the interfaces, but for me the installer didn’t put it in the start menu; I had to dig in the program files folder to find the .chm file manually.  The trick was figuring out which programming language was best suited to the task of interfacing with the DLL and doing manipulation of config files in formats like INI and JSON.  The setup would have to be totally portable–we run our tools from a file share on our server and we can’t just go installing random runtimes on customer computers.  It also had to work well on Vista, 7, and 8.x, which makes things like PowerShell difficult since the varying versions provide different functions (e.g., PowerShell 1.0 doesn’t have native JSON support).  Recently my language of choice has been Python, and exploring that option is how I found what turned out to be a huge life saver: Portable Python.

Portable Python is exactly that: a portable python environment that can run from a local folder, UMS device like a flash drive, or a network file share.  Additional modules can be installed with relative ease, and it works on all the operating systems we support right now.  Python has lots of great modules for file management, file manipulation, config parsers for INI and JSON, pretty much everything I need.  Nicely enough, one can also easily call functions exported from a DLL file using an interface called ctypes.  They really mean for you to use AutoItX via COM in python, but that requires AutoItX to be installed locally which we aren’t going to do.

So here’s my setup: portable python and my scripts directory are stored in a file share on our server.  It’s easy to build a batch file that executes the portable python interpreter, passing my scripts as command line arguments to get them running without doing messy file-association mods on the customer PC’s.  The AutoItX DLL is also being hosted on a file share and my python script can copy that to a local folder then manually load it using ctypes.  Here’s an example of one of my scripts:

Please note that the big hurdle that I had to cross was related to Unicode strings.  At first I was just passing regular strings to the AutoIt functions (like WinWait) and they never matched any windows no matter what.  After some digging I found that AutoIt is expecting Unicode strings and it is assuming that all strings passed in are already Unicode and interpreting them as such.  Explicitly passing all strings as Unicode fixes that problem.

Advertisements

UEFI, SecureBoot, PXE, and You

For a while now we’ve had a need to PXE-boot computers that are set up for UEFI and SecureBoot but haven’t quite been able to pull it off.  For a long time, information on the subject was really difficult to come by and was mainly in the form of discussions by experts in the process of research and development.  I’m not an expert in the field of…anything really; I’m just an everyday computer repair grunt who knows enough about a lot of different things to make him wish he knew a lot about any one thing.  Over time I’ve occasionally taken the time to do a little more searching to see if tutorial-format information had become available and today I was not disappointed.

I came across this page on the Ubuntu wiki, and it was the glue I needed to finally make the pieces fit together in my mind.  It didn’t work exactly as described, so I’ll document my actual steps here.  I’m starting off with an already existing PXE environment that was working the old way, e.g. BIOS booting using pxelinux.  This is one of the big differences between my setup and the Ubuntu wiki page and that’s the starting point for this tutorial.  There are a lot of great resources out there for that so if you need help it’s only a web search away.

One thing to keep in mind is that my server is running CentOS, but my bootable PXE environment is LinuxMint.  What that basically means is that the config files will be CentOS related but I’m borrowing files from Ubuntu (since they have signed bootloader files easily documented in their tutorial).

First, here’s a list of files as per the Ubuntu wiki page:

  • shim.efi.signed from the shim-signed package, installed as bootx64.efi under the tftp root

  • grubnetx64.efi.signed from the grub2 source package (and shipped in the grub-efi-amd64-signed binary package), installed as ‘grubx64.efi’ under the tftp root

  • unicode.pf2 from the grub-common package, installed as grub/fonts/unicode.pf2 under the tftp root.

So getting these files is pretty easy.  We’re just going to extract them directly from the packages they belong to.  Now Ubuntu has a script on their page that is supposed to do this stuff, but I want to do it manually.  For one thing they have you grab grubnetx64 from the Saucy repo but that did not work for me.  It would load the grub menu, then work intermittently, otherwise giving two errors: “couldn’t send network packet” and “you need to load the kernel first”  Doing some searching it appears there have been bugs in that file fixed recently and the one from Trusty worked for me fine.  Here’s what we want to do (I ran these from my existing BIOS PXE environment):

apt-get download shim-signed
ar vx shim-signed_1.6+0.4-0ubuntu4_amd64.deb
tar -xvJf data.tar.xz
cp ./usr/lib/shim/shim.efi.signed ./bootx64.efi
# !! now you're ready to copy ./bootx64.efi to your tftproot
rm -rf ./usr data.tar.xz control.tar.gz debian-binary shim-signed_1.6+0.4-0ubuntu4_amd64.deb

wget -O grubx64.efi http://archive.ubuntu.com/ubuntu/dists/trusty/main/uefi/grub2-amd64/current/grubnetx64.efi.signed
# !! now copy ./grubx64.efi to your tftproot

apt-get download grub-common
ar vx grub-common_2.02~beta2-9ubuntu1_amd64.deb 
tar -xvJf data.tar.xz
cp ./usr/share/grub/unicode.pf2 ./
# !! now copy unicode.pf2 to your tpftproot under grub/fonts (e.g. /tftpboot/grub/fonts/)
rm -rf ./usr data.tar.xz control.tar.gz debian-binary grub-common_2.02~beta2-9ubuntu1_amd64.deb

Now we need to create a grub configuration file that will be stored on tftproot under the “grub” directory (e.g. /tftpboot/grub/grub.cfg).  Mine is a bit different from the one on the Ubuntu wiki since I’m mounting an NFS root and they were not.  My NFS root is on my server (192.168.1.15) under /exports/nfsrootqiana.  Keep in mind that in the kernel load lines , “(pxe)/” refers to the tftp root directory and that’s where the files vmlinuz-3.15.3 and initrd.img-qiana are located.  So here’s my config file:

# /tftpboot/grub/grub.cfg
set default="0"
set timeout=-1

if loadfont unicode ; then
 set gfxmode=auto
 set locale_dir=$prefix/locale
 set lang=en_US
fi
terminal_output gfxterm

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
if background_color 44,0,30; then
 clear
fi

function gfxmode {
 set gfxpayload="${1}"
 if [ "${1}" = "keep" ]; then
 set vt_handoff=vt.handoff=7
 else
 set vt_handoff=
 fi
}

set linux_gfx_mode=keep

export linux_gfx_mode

menuentry 'Linuxmint Qiana' {
 gfxmode $linux_gfx_mode
 linux (pxe)/vmlinuz-3.15.3 $vt_handoff root=/dev/nfs initrd=initrd.img-qiana nfsroot=192.168.1.15:/exports/nfsrootqiana ip=dhcp rw
 initrd (pxe)/initrd.img-qiana
}

We also need to tweak our DHCP config to respond to UEFI PXE requests.  Here is my entire updated config, including the lines that make the old BIOS PXE boot work:

# /etc/dnsmasq.d/dhcp.conf
dhcp-range=192.168.1.100,192.168.1.254,12h
dhcp-option=3,192.168.1.1
dhcp-option=15,alltech.local
dhcp-boot=pxelinux.0
dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-boot=tag:efi-x86_64,bootx64.efi
except-interface=wan0
dhcp-authoritative

As you can see, we’re using dnsmasq for DHCP.  On our old CentOS (5.10), the builtin dnsmasq didn’t work with that “tag:” syntax so I had to update the program.  The source version of dnsmasq doesn’t come with the init scripts for Red Hat so I had to cheat a little bit.  I built dnsmasq from source and did a “make install” like usual.  Then I simply edited /etc/init.d/dnsmasq and changed it like so:

# /etc/init.d/dnsmasq
# change...
dnsmasq=/usr/sbin/dnsmasq
# to...
dnsmasq=/usr/local/sbin/dnsmasq

Now it will happily use the nice new version of dnsmasq. No fuss, no muss.

We should now be able to boot PXE from either a BIOS motherboard or a UEFI motherboard with or without SecureBoot enabled. For the curious, here’s the sequence of events as I understand it. Please correct me in the comments if I get something wrong:

  1. You tell the computer you want to boot via PXE.  It sends out a PXE request.
  2. DHCP Server initially sets the boot image file to pxelinux.0
  3. If booting from UEFI, the DHCP Server sees that the client architecture is 7 (EFI) and sets the tag “efi-x86_64”.
  4. If the efi-x86_64 tag is set, the DHCP Server switches the boot image to bootx64.efi (otherwise, the PC boots from pxelinux.0)
  5. DHCP Server sends the response to the client
  6. (from here on, I’m following UEFI sequence) The client requests the file bootx64.efi via TFTP
  7. SecureBoot checks the signature on bootx64.efi, and it has a valid Microsoft signature
  8. The firmware then loads and runs bootx64.efi
  9. bootx64.efi looks for grubx64.efi via tftp and checks its signature.  It’s signed by Canonical, and passes the check
  10. bootx64.efi loads and runs grubx64.efi, which in turn loads the grub config from tftp
  11. Normal grub boot sequence occurs, and we’re cooking with gas

Introducing: Firefox Extension Killer

At the shop we generally recommend our customers to use any browser except Internet Explorer.  This probably doesn’t come as a surprise to anyone who has spent any amount of time fixing the things people ruin on their computers or anyone who has ever wondered why their markup/css just doesn’t work right in one browser when it works right in every other browser (or why it only loofirefox_logo-only_RGBks right in one browser!).  Because of it’s ability to behave more like the browsers of yesterday when configured as such (which is EXACTLY what our older customers want) we generally prefer Firefox.  As a result, cleaning up malicious addons has become an everyday chore for us when people bring in junked-up computers.

Out of the box, cleaning up extensions in Firefox is kind of a mixed bag.  Sometimes an extension will have a remove button, other times it won’t.  Why is this?  If an addon was installed via the AddOns Manager, it will have a remove button.  If the files were installed manually from outside Firefox, there will not be a remove button.  I wanted to link to the area of the Mozilla knowledge base where they explain this decision (which I read years ago), but I can’t seem to find it anymore.  I think the gist was that since a manually installed addon could have files that aren’t tracked by Firefox, they didn’t want people to think that removing the extension in Firefox’s AddOns manager would remove all files associated with the addon.

Of course what this has led to is that most malware addons will install manually, leaving the typical user with no way of removing them from Firefox.  Mozilla has an article that explains how to remove them manually, but your average Joe is never going to be able to go through this process.  Even if one is technically inclined enough to follow the directions it’s extremely tedious to look in so many places and it’s time consuming to boot, which is why I put together the Firefox Extension Killer.

I have actually tried to write this program several times.  At first I wanted to do it in C++ because of its lack of lots of system dependencies (aside from any libraries that are used, of course).  That was a hard thing to commit to considering my hatred for the Win32 API, but I just felt like it was the best choice at the time.   I’d thought of something like C# which is much more elegant on Windows than C++, but when I first started working on the tool we were still doing a lot of Windows XP machines and they don’t all have the .Net Framework installed; I just couldn’t see having to spend 10 minutes installing .Net 3.5 to save 5 minutes of repair time.  Then it came to the development tools on Windows: either Visual Studio Express which is a huge resource hog, or a cobbled together environment trying to emulate my beloved Linux work-flow.

When that got toangry-codero cumbersome to deal with, I turned to cross-compiling from Linux to Win32.  Years ago successfully cross compiling code was similar to building a running internal combustion engine out of Lincoln Logs, but nowadays there are full tool-chain sets in the Ubuntu/Mint repos that “just work” after installation, even if the MinGW version names are unbearably confusing.  What this work-flow meant though was that testing the registry access parts of the code would be impossible in Linux.  I had the idea of writing some wrapper functions and implementing a virtual registry testing library for C++ (the perfect textbook solution) but very quickly realized that writing a library that could interact with and emulate the Windows Registry with any amount of configurability would take a LOT longer than the whole rest of Firefox Extension Killer.

After going through all this, I became very quickly disenfranchised.  I had at least gotten a CLI tool running that looked in (most of) the addon locations and just removed everything without any options.  This worked OK, since we mostly only wanted Adblock Plus installed and that was an easy reinstall.  Apart from that, I quit working on it for a year or so; it was just too painful.

Recently things have been slow at the shop, and I really started thinking about it again.  I’d piddled with a few designs on and off over time, but nothing really seemed to fit right.  The playing field is different now too; we’re pretty much working on Vista and higher now which has at least  .Net 3.0 built in.  It occurred to me that what was not an option before was the best option now: C# with Windows Forms.

My application design skills are the worst.  I do not exaggerate.  I realized recently that the designs I layout before coding are at least as bad as the stuff I come up with when I just start typing, so I installed SharpDevelop in a Windows VM and just started typing.  Two days later I had a release of Firefox Extension Killer.

So head on over to GitHub and check it out (or clone it out, as it were).  Right now I only provide a SharpDevelop project file as a means of building it, but it looks like SharpDevelop squirted out a Virtual Studio project file too.  I didn’t ask for that, but I left it in the repo anyway in case it works.  YMMV.

Dropping Google Calendar – Webdav to the rescue

There has been a lot of uproar recently about Google’s new (some say lack of) privacy policy.  I’m not really going into the details of that here because frankly, it doesn’t have much to do with my personal decision to move away from their services.  For the last few years I’ve been nervous about how large and monopolistic Google has become and I’ve had the idea in the back of my mind that I might not want to stick with them for much longer; the new privacy policy announcement was really just the little shove I needed to get started with the process.

At the shop (aside from email, of course) the biggest potential problem for us is calendaring.  We’ve been very happy with how nicely Google’s calendars work in our ability to share them with each other and view them on various devices like Android phones.  I knew that to replace that I would need equivalent functionality or we’d really be moving backwards in our scheduling workflow.

I tried a couple of products with limited success:

  • DaviCal – I couldn’t really even connect properly to this in the first place even after reading several howto’s
  • CalendarServer (also known as Darwin Calendar Server) – it would initially, but seemed to flake out a lot (suddenly not serving pages in the web based interface, etc.)

Then I realized that pretty much any client that supports third party calendars can handle CalDav/WebDav.  This seemed to be my golden ticket, as WebDav is pretty easy to set up and is widely used—meaning that show-stopping bugs aren’t likely to live long.

At this point it is actually not a simple thing to use WebDav or anything else besides the Google Calendar on Android phones, so there may be a future post on that. Since this is the case no matter which method we choose it’s not really a deciding factor here.

What is WebDav?  The simplest explanation is that it’s a protocol that allows read/write access to files over a network as an extension to the HTTP.  As far as I can tell by reading online (though I haven’t seen much on the topic without reading RFC’s), the only difference between CalDav and normal WebDav is that it minimizes the amount of data that must be transferred during a calendar sync.

The Apache webserver has a widely supported module called mod_dav that adds WebDav support and that’s what I chose to use.  There is a mod_caldav module but it doesn’t look like it’s kept up very well.  As of this writing it hasn’t been modified since March of 2010, nearly two years ago.

On our CentOS server, things were pretty simple once I got through it.  I found a few hotwo’s, none of them complete.  Here’s what I did:

  1. Our virtual hosts are set up by adding VHOST_hostname.conf files to /etc/httpd/conf.d/ so I added VHOST_webdav.conf like so (the mod_dav module was already enabled for us, but you can enable it in /etc/httpd/conf/httpd.conf if not):
    <IfModule mod_dav.c>
        DavLockDB webvar/DavLock
        Alias /webdav "/var/www/webdav"
        <Directory /var/www/webdav>
            Dav On
            Options +Indexes
            IndexOptions FancyIndexing
            AddDefaultCharset UTF-8
            AuthType Basic
            AuthName "Alltech WebDAV Server"
            AuthUserFile /etc/httpd/webdav.users.pwd
            Require valid-user
            Order allow,deny
            Allow from all
        </Directory>
     </IfModule>
  2. Now you notice that line that says “DavLockDB webvar/DavLock”?  That means that mod_dav will create it’s lock file in {ServerRoot}/webvar with the basename DavLock for the files.  In our case {ServerRoot} is /etc/httpd, so we need to create that directory and give apache write permission for it:
    mkdir /etc/httpd/webvar
    chown apache:apache /etc/httpd/webvar
  3. We’ll also need to create the folder that our Directory container referenced (which is where our files will live):
    mkdir /var/www/webdav
    chown apache:apache /var/www/webdav
  4. Of course we’ll need to create a standard htpasswd file for the users and passwords.  According to our config, we want to create it thusly:
    htpasswd -c /etc/httpd/webdav.users.pwd myusername
    It will prompt you for a password for the user “myusername”.  To add more users, just leave off the “-c” parameter and substitute the new user name.
  5. Now restart apache:
    service httpd restart
  6. You can test it by using the “cadaver” command-line webdav client:
    cadaver http://servername/webdav
    If you don’t get errors and you can do a directory listing with ‘ls’, you win.

Anti-theft protection in a montor? Really, Gateway?

Recently at the shop we’ve been needing a few new monitors.  Instead of just buying new ones, we went to our local scrap guy to see if he had any LCD monitors that looked repairable; mostly we just fix the ones with bad capacitors in the power supply, which turns out to be a pretty high percentage.

We fixed 2 Dell monitors right off and I’m using one of them right now (score!).  The problem came when we tried to fix a really nice (well, it looks nice) Gateway 19″ wide screen.  It had 9 blown caps so I ordered those and soldered them in.  No problem.  It fired right up and looked beautiful…for an hour.

It turns out this monitor has a built-in anti-theft system.  Why you need an anti-theft system in a monitor, I have no idea.  It looked like we were going to be able to get it unlocked for us by Gateway support but they proved to be even worse at their jobs than you normally expect from outsourced tech support.

The problem stems from the time-frame in which the unlocking must occur.  The monitor will generate a 3 digit “challenge code” that Gateway uses to generated the unlock code, but the challenge code times out after 1 hour and a new one must be generated.  The guy at Gateway technical would take our challenge code and come back 12-24 hours later with an unlock code for us.  After 2 or 3 times of trying to explain to this guy that he had to respond within an hour for it to work I tried opening another ticket.  That technician basically said “go away”.

So now I have $19 and an hour of labor in a monitor that works, but I can’t use.

Moral of the story?  Buy a Dell.