Category Archives: Hardware

Patching ScaleIO 2.0 VMware Hosts

Recently did an ESXi patch run, along with some BIOS and firmware updates on a ScaleIO 2.x environment (more precisely 2.0.5014.0). The environment consists of some Dell PowerEdge servers, some of which are ESXi 6.0 build 3380124, some are Linux based, non-virtualized hosts. Luckily this environment was ScaleIO 2.x, because this version has a real maintenance mode in it (1.3.x did not). This means that while I can only patch one host at a time in this layout, I can do it fairly quickly and in a controlled fashion.

ScaleIO Maintenance Mode vs. ESXi Maintenance Mode

These are, obviously, two different things. With ScaleIO maintenance mode, you can put one SDS (providing storage services) host (at least in this configuration with two MDM’s) at a time into maintenance mode, which does not have an adverse impact on the cluster. The remaining SDS will take care of operations, provided it too does not break or go down at the same time.  After you are done patching, you exit maintenance mode, which the makes sure all changes are rebuilt and synced across the cluster nodes. This takes some time depending on the amount of data involved.

ESXi maintenance mode on the other hand, deals with putting the VMware hypervisor layer into maintenance mode so you can patch and perform other operations on it with no VMs running. The order is:

  1. 1. ScaleIO
  2. 2. VMware ESXi

And when coming out of the maintenance break, it’s the reverse.

I left the SVM (virtual machine on the host which takes care of the different functions that the host has, technically a SLES appliance) that I was patching, but I powered it down gracefully before putting the host into maintenance mode.

So accounting for all these things, my order was:

  1. 1. Migrate all running VMs except the SVM off of the host using vMotion
  2. 2. When the host is empty (bar the SVM), put ScaleIO into maintenance mode
    1. a. This is done via the ScaleIO GUI application, on the Backend page, by right clicking on the host. I did not have to use the force option, and neither should you…
  3. 3. Shut down the SVM via “Shut Down Guest” in vCenter
  4. 4. Put the host into maintenance mode without moving the SVM off the host (I suppose you could move it, but I didn’t)
  5. 5. Scan and Remediate and install other patches (I installed BIOS, iDRAC and some other various updates via iDRAC; I had set them to “Install next reboot” so they would be installed during the same reboot as ESXi does remediation)
  6. 6. Once you are satisfied, take the host out of maintenance mode
  7. 7. Start the SVM on that host
  8. 8. Wait for it to boot
  9. 9. Exit ScaleIO maintenance mode (see 2.)
  10. 10. Check to see that rebuild goes through (ScaleIO GUI application, either the Dashboard or Backend page)
  11. 11. Make sure all warnings and errors clear. During host remediation and patching, I had the following errors:
    • High – MDM isn’t clustered (this is because you’ve shut down one of the SVMs containing the MDM role)
    • Medium – SDS is disconnected (for the host being remediated)
    • Low – SDS is in maintenance mode (for the host being remediated)
  12. 12. After the SVM starts, it should clear all but the last alert, and once you have Exited Maintenance Mode, the final alert should clear
Exiting maintenance mode in ScaleIO GUI application
Rebuilding after exiting maintenance mode in ScaleIO

(Expected) Alerts during maintenance

As mentioned, you will have alerts and warnings during this operation. I had the following:

  • First, when putting the SDS into maintenance mode in ScaleIO, one warning about SDS being in maintenance mode:
SDS still on, ESXi not in maintenance
  • After SVM is shut down and ESXi is also placed in maintenance, two more:
All three alerts after host is in maintenance and SVM has been shut down
  • Then once you have remediated and taken the host out of maintenance, and started the SVM, you’re back to one, as in the first picture.
  • When you take the SDS out of maintenance, it will clear the last alert

Note that the highest rated alert, the Critical “MDM isn’t clustered” is actually noteworthy. It means that the SDS you are taking down for maintenance had the MDM role (critical for management of ScaleIO). Normally you’d have another one, and you shouldn’t proceed with any of this if you can only find one MDM, or if you already had this (or any other alert).

EMC has this to say about MDM’s (also see the document h14036-emc-scaleio-operation-ensuring-non-disruptive-operation-upgrade.pdf):

Currently, an MDM can manage up to 1024 servers. When several MDMs are present, an SDC may be managed by several MDMs, whereas, an SDS can only belong to one MDM. ScaleIO version 2.0 and later supports five MDMs (with a minimum of three) where we define a Master, Slave and Tie-breaker MDM.

Roles / Elements in ScaleIO

You can see the installed roles in VMware in the notes field, like so:

Roles in the Notes field in VMware

Elements or roles are (may not be a complete list):

MASTER_MDM – Master MDM node, Meta Data Manager, enables monitoring and configuration changes
SLAVE_MDM – Secondary MDM node, will take over if Master is unavailable
SDS – Storage node, ScaleIO Data Server, provides storage services through HDD, SSD, NVMe etc.
SDC – ScaleIO Data Client, consumer of resources (e.g. a virtualization host)
RFCACHE – Read-only cache consisting of SSD or Flash
RMCACHE – RAM based cache
LIA – Light installation agent (on all nodes, creates a trust between node and Installation Manager)
TB – Tiebreaker, in case of conflicts inside cluster, counted as a type of MDM, non critical except in HA/conflict situations

ESXi funny business…

While running remediate on the hosts, every single one failed when installing patches.

Scary Fatal Error 15 during remediation

A very scary looking Fatal Error 15. However, there’s a KB on this here.

So, (warm) reboot the host again, wait for ESXi to load the old pre-update version, and do a re-remediate without using the Stage option first. I used stage, as I’m used to, apparently this breaks. Sometimes.

And to re-iterate, I was patching using vCenter Update Manager (or VUM) from 6.0 build 3380124 to 5050593.

Sources from (not actually for the version in use, but similar enough in this case. Use at your own risk..

ScaleIO v2.0.x User Guide.pdf contained in the above mentioned

Home Lab Xeon

The current home lab setup consists of an Intel Core i3-2100 with 16GB of DDR3, a USB drive for ESXi (on 6.5 right now) and a 3TB WD for the VMs. While the Intel i3 performs perfectly for my needs, I came across a Xeon E3-1220 (SR00F, Ivy Bridge), which should be even better!

For the specs, we have the following differences:

Model Intel Xeon E3-1220 Intel Core i3-2100
Released: Q2-2011 Q1-2011
Manufacturing process: 32nm 32nm
Price originally: 189-203 US dollars (more in euroland) 120 USD
Core count: 4 Cores 2 cores
Hyperthreading No Yes
Base Freq: 3.10 GHz 3.1 GHz
Turbo Freq: 3.40 GHz No
TDP: 80 W 65W
Max Memory: 32 GB ECC DDR3 32 GB Non-ECC DDR3
L1 Cache: 128 + 128 KB 64 + 64 KB
L2 Cache: 1 MB 512 KB
L3 Cache: 8 MB 3 MB

So we can see that the Xeon part is 4 core processor, without hyperthreading, so real cores as opposed to the i3’s threads. It’s more power hungry, which is to be expected, but can also Turbo at a higher frequency than the i3. Also, the Xeon has more cache, which is also to be expected with a server grade component.

A notable thing is that the Xeon, being a server part, does not include the GPU components, so I’ll have to add a GPU at least for the installation. I run the server headless anyway, but I want to see it POST at least. I think I’ll have to add a PCI card for this it has no PCI slots so, as I only have one PCIe slot (well there are some x1 slots but I have no such cards), and that’s used by the NIC. The motherboard is an Asrock H61M-DGS R2.0 which has one x16 slot and one x1 slot. Maybe I’ll do it all headless and hope it posts? Or take out the NIC for the installation?

Some yahoo also tried running an x16 card in an x1 slot here. Might try that but since I have to melt off one end of the x1 slot, probably not.

There are apparently some x1 graphics cards, but I don’t have one as I mentioned. An option could be the Zotac GeForce GT 710, which can be had for 60 euros as of this post.


I went to the pharmacy to get some pure isopropyl alcohol. It wasn’t on the shelf, so I had to ask for it. I told the lady I need some isopropyl alcohol, as pure as possible. She looked at me funny and said they had some in stock. I told her I’m using it to clean electronics, so she wouldn’t suspect I’m some sort of cringey soon-to-be-blind  (not sure if you get blind from this stuff, but it can’t be good for you) wannabe alcoholic, to which she replied that she doesn’t know what i’ll do with it, or how it will work for that. She got the bottle, which is described as “100 ml Isopropyl Alcohol”. There is a mention of cleaning vinyl disks and tape recorder heads on the back, so I was vindicated. There’s no indication of purity on the bottle, but the manufacturer lists above 99.8% purity here. Doesn’t exactly match the bottle, but it’s close.

Why did I get isopropyl alcohol? Well, because people on the internet said it’s good for cleaning off residual thermal paste from processors and CPU coolers. With common sense 2.0, I can also deduce that anything with a high alcoholic content will evaporate, and not leave behind anything conductive to mess things up. Oh and it cost 6,30€ at the local pharmacy. It’s not listed on the website (or it says it’s no longer a part of their selection).

Let’s see how it performs. I’m using cotton swabs, but I suppose I could use a paper towel. If it leaves behind cotton pieces, I’ll switch to something else.

The Xeon originally had a passive CPU block and a bunch of loud, small case fans, but I will use the same cooler as for the i3.

Take out the i3 and the cooler. Clean the cooler off with the isopropyl:

Isopropyl worked wonders

Put in the E3, new thermal paste. I used some trusty Arctic Silver 5.

Termal paste added, note artistic pattern

Re-attach the cooler and we’re off to the races. I’ll note here that I hate the push through and turn type attachments of the stock Intel cooler. Oh well, it’ll work.


Powering on

Powering the thing on was the exciting part. Will there be blue smoke? Will it boot headless? Will it get stuck in some POST screen and require me to press a button to move on? Maybe even go into the BIOS to save settings for the new CPU?

Strangely enough, after a while, I started getting ping replies from ESXi meaning the box had booted.

There’s really nothing left to do. ESXi 6.5 recognizes the new CPU and VMs started booting shortly after.

Xeon E3 running on ESXi 6.5

Flow of things

A very long time has passed since I last posted anything. In that time, I’ve done an ass ton (metric, in imperial that’d be approximately 45/64th’s of one quarter cup liquid ounce of.. inches?)  of work, been to Switzerland and back, had my son start elementary school, and various other bits and bobs. Maybe that’s why? Anyway, I’ll start rambling off things that come to mind.

So I went to Switzerland, Geneva to be more exact. And to be even more exact, I visited CERN! The inner geek in me is still excited. That place is, to put it bluntly, amazing. We started by checking in at the visitor center, where we got our badges. I took the opportunity (at the recommendation of one of our hosts), to visit the gift shop and pick up a t-shirt and coffee mug. The mug has the four component formulas for, well, everything important, i.e. the Standard Model Lagrangian. Don’t ask me to explain it, because I’m pretty sure I couldn’t. The t-shirt I can explain. Not only was it made somewhere in Asia, but it also has on it the original Tim Berners Lee proposal for the world wide web. The back has his boss’s comment “Vague, but exciting”, on it. Both items are in frequent use.

At CERN, I visited the control room for ATLAS, one of the experiments using the large hadron collider. The LHC itself was being upgraded to allow for higher energy level collisions in the future. Pity we couldn’t visit the actual detector, or see the actual uhm.. tube where the particles travel in a circle before hitting each other every once in a while. We also visited the computer center.  As a computer guy, I was pretty darn impressed. The amount of hardware that’s in there is staggering, and the connections to the outside world are even more impressive. I was told there wasn’t “much” science going on, and still the aggregate bandwidth of connections to and from the facility and to research facilities around the world was at over 7 GiB, with over 200 000 running jobs. They told us it gets to around 13-15 GiB when there’s a real buzz. There was a nifty touch screen in the lobby of the computer center, built around google earth, that you could spin around to see the different connections around the globe. Finland’s share? A meager 0,3% of the computing being done. Meh. The lobby also had some display cases with various old hardware: old modems, fiber optics, hard drives and so on.

Geneva was a nice place in general. The climate was nice, the views spectacular and the people generally very nice. I had that same nagging feeling that I had in Paris in 07, where the French speaking people were just acting.. weird. We had a waiter that was muttering something under his breath the whole time he was serving us. There was that same air of arrogance and displeasure at having to speak English. The hotel was a refreshing exception (as it was in Paris), and I can easily recommend it for anyone looking for a reasonably priced hotel in Geneva. We stayed at, *drumroll* the Hotel de Geneve! Located at 1, pl. Isaac Mercier, Geneva 1201, Switzerland, it seemed to be a fairly central location. It was a short 10 minute walk from the train station, and not far from the river for instance.

On our second day, we took the train to Lausanne. I had perch. Nice expensive looking place by the shores of Lac Léman (Lake Geneva). The train ride was maybe an hour, or a little less and very smooth. Saw an Aston Martin Vanquish drive by. The whole place seemed to be in a perpetual slow motion, and somehow at ease or at rest. Didn’t really see much of the city, we just had lunch, but what little I saw was nice.

The journey back was eventless, if it wasn’t for the small incident at the airfield in Geneva. We were taken to our plane (an Embraer 190) by bus, and had to wait outside the plane for a considerable amount of time as the idiots piled into the plane (how hard is it to just find your place, and stow your luggage?). While waiting, I figured I’d take a few pictures. I took a picture of one of my traveling companions, with the plane in the background, and then turned around to take a picture of the scenic mountains that basically surround the whole place. At this point, one of the yellow vested… whatever she was, told me to put the phone away. No pictures! Put it your bag! I told her there were no signs posted anywhere that I couldn’t take a picture, but she would have none of it, and I yielded, putting my phone in my pocket.

Now, I am aware that standing on the tarmac, there is in theory a risk that something will happen that requires my attention. On the other hand, if a plan lands on us, I doubt I would have time to do some kind of Die Hard-type jump to safety, phone or no phone. There were also no spinning propellers that I could accidentally walk into. I think there was even a small roped fence thing preventing us from wandering onto the runway or other areas around the plane.

I was not given any reason for why I couldn’t take a picture. This always irks me. If there is no sign prohibiting photography, or an announcement, and I have used my common sense to assess that taking a picture does not pose a risk to my or anyone elses health, I’m going to take pictures. I have no reason to fight with airport people. They are doing their job. I still fail to see how my photography could cause any harm. Also note, the queue into the plane was *not* moving, so I was not holding up the plane, telling everyone “hold on, I need to tweet this shit!”.

“Is this not a reasonable place to park?”

Enough about travel again! Seems I can’t get enough of it. Later this year, though, I’m flying over to Edinburgh, which might be the place to be now that they are voting for independence! I might get a chance to visit the newest independent country in the world. Or maybe not, in case the No-vote is the winner.  The vote might be today?

On the hardware side of life, I’ve been doing some upgrades for my backup and storage infrastructure. For local onsite backups, I now have an Iomega IX2-200 (cloud edition), with twin 3TB Western Digital Red drives, in RAID1. It’s not the newest or the fastest NAS out there, but it works. On my main workstation I have replaced the previous 2x1TB RAID1 set with a 2x2TB RAID1 set. Just added one terabyte. I now have a bunch of spare 1TB disks, which will probably be incorporated into a FreeNAS build I’m working on. I had some issues trying it out earlier this month, but I think it was just Samba misbehaving. It would disconnect in the middle of a file transfer, and tell me the path is not accessible. According to FreeNAS, things were a-ok. It’s not like I’m a FreeNAS guru or anything, so I’ll have to put in more hours to that build to get it working. It might end up being up to 8x1TB. Currently I have only 8GB of RAM (ECC, though), but I’ll probably want to upgrade that to at least 16, maybe even 32. The thing is, that means I have to get a different motherboard, processor and.. Oh well.


Lenovo Thinkpad T440s – 6 months in

I’ve now had the Lenovo Thinkpad T440s as a work machine for the past 6 or 7 months. Here are some short observations, things I like, things I don’t like, things the broke, etc.

Things I do not like:

  • The gorram touchpad. Get it out of here! Horrible the way there is like a single button (the size of the entire touchpad), and a certain area for the right mouse button etc. Just unusable in my opinion
  • The keyboard used to be better… now it resembles something that comes from Cupertino, and is not as comfortable to use as the previous thinkpad-y keyboards
  • No more nipple buttons! How am I supposed to use the trackpoint (a.k.a. the nipple) without the two buttons below the keyboard? I’m not, that’s how! External mouse is basically an absolute necessity
  • They’ve slimmed it down so much the keyboard leaves marks on the screen when you have the lid closed. It’ll only get worse, and I hope it doesn’t permanently damage the screen. I do have a screen-filter in between so hopefully that protects the LCD slightly.

Things I like:

  • Screen is great. 14″, FullHD (yes, It’s not 1440p or whatever). You could get it with either a touchscreen or not. Obviously mine isn’t a touchscreen, as I was buying a laptop, not a tablet
  • 256GB SSD. Not the fastest out there, but I like
  • Connectivity. With the docking station, I have enough ports to fill my needs. USB 3.0, 2x Display Port (which I have connected to my two external Dell screens), etc. etc. I’ve missed the optical drive a few times. But not enough to get an external drive to lug around
  • The overall form factor, size and weight

Things that have broken or failed more than once, or annoyed me

This list is longer that I would like. Compared to previous Thinkpads that I have used or owned, this is unusual

  • SSD. Started failing when I was saving files (for instance), and eventually stopped being detected at boot. Replacement was sent by Lenovo, and I swapped it out. In hindsight, do not do this on your own. The case is a bitch to open. Get their onsite tech to do it.
  • Keyboard. Broke a button while fiddling with it. A piece of plastic came off and the button was forever broken. My fault entirely. Ordered a replacement keyboard, swapped it out. Easier than the SSD. A bit harder than some Thinkpad models in the past.
  • The piece of metal that keeps the ethernet cable in place! This is incredibly annoying. For some reason, the ethernet cable doesn’t *click* into place anymore. Something is missing. Not sure this is a warranty thing. I’ll just survive, I think. I use it in the dock about 70% of the time anyway
  • Issues with the external screens, when docked. I have two Dell U2713HM screens attached via Display Port cables to the dock. Randomly, the screens will go blank, even when the laptop is securely seated (and locked) into the dock. Sometimes resolutions get messed up, so that one screen has a lower resolution. This might be a Windows 8.1 issue too, but still, annoying. Issues waking up from sleep, or power save
  • Serviceability. I wish it was easier to open the case. Granted, I don’t have to do it. I can get their onsite or whatever to do it. But I liked how you could open the slot for memory, or the hard drive, or whatever, and not have to rip the entire case to bits. Screws are also not enough, there are plastic clips that *will* break if you are not careful when opening the case. I wish it was more like my T410s, where everything, more or less, was behind it’s own hatch and/or easily replaceable
  • Not available with more then 12 GB memory. Why? Why the I7 processor, but then limit the memory to 12GB? Doesn’t make sense in 2014…

Not sure I can recommend this laptop. There are a lot of annoying things with this machine. When docked, it works mostly great, and with the 256GB SSD, I7 processor, and with it’s dual DP ports supporting large external screens, it is a powerful rig. But a lot of annoying issues. Not sure what I would get, if I didn’t get this one. Apple is out, never liked HP.. what other business type machines are there that I would like? Dell? Always thought they were a bit clunky.. I dunno.

Observations from an ebook noob

I’ve been the owner of an ebook reader (see the previous post) for all of two weeks now. I have used my kindle nearly daily, and it’s a handy thing to have around. So far, I’ve mostly been reading issues of Linux Journal (who moved to a digital format two years (?) back), the scifi book by MK Wren that I mentioned, and then various tests.

But about the medium. Surprisingly, I fucking hate that there are format restrictions, DRM and all that jazz. Why have two formats that do essentially the same thing on different devices? Profits, probably. Businessy stuff that I don’t understand. There are of course, ways around things like this. I read somewhere that you can root a Kindle, which then enables functionality not found on the retail device. There are various converters for formats, such as Calibre, which enables management and conversion between formats. I have read that the Kindle (un-rooted?) will not eat stuff that has been un-DRM-ified using a converter, or that it will read books that have been converted at all. I haven’t tried the software yet, so I’ll have to get back to you.

The issue of DRM is a difficult one. I do not believe in crippling content and/or software. Your product should be good enough so that people want to pay for it. And I will. The amount of money I spend on software, movies and music in a given year is not a small one. We own several shelves of music, several gigabytes of digital music, and probably in the neighborhood of 500 DVDs and Blurays. I prefer FLOSS, but if it doesn’t do what I need it to do, I’ll probably buy something. I own my copies of Windows, on all of my hardware. And so on. Ok, disclaimers aside, the point I was trying to make is: If your content is good and there is a need for it, people will pay for it. DRM will never be an effective solution, ever. People will always find a way around it.

Okay, done venting!

I’m still miffed that I can’t read my technical manuals or whitepapers, which are in PDF format, on my Kindle. I would really find it useful if I could carry that with me when I go on consulting gigs, so I could pull up any number of manuals when I’m in a server room somewhere doing an install. Yeah, I can use a laptop, but that will run out of battery on most install gigs, and it’s not comfortable to have when you’re behind a rack for instance. Printing them is also out of the question, as they might be hundreds of pages. This is really a use case I can get behind, though, I do admit it is a comfort thing, more than a necessity for me.

I ran into that pesky “out of memory” message, trying to read a tiny 15 MB pdf. I don’t get it. Surely the device has more than 15 megs of RAM, and I hope it doesn’t cache the entire document when you read it. Maybe a slight read-ahead and read-back? Conversion might be the answer here, but, as I said, I will have to get back when I’ve tried it.

As for the content: I have not bought anything from Amazon yet. I have bought The Book of PF (3rd edition) from No Starch (really like their stuff!), some indie content, and then the scifi books through.. whatever it was. Paid by paypal or credit card, then transfered them through the USB to the Kindle. Works fine.

There is in-device buying. I’ve seen ads for $1.99 books on the Kindle, and sooner or later, I’ll click on one. It will be interesting to see if there are regional restrictions on that. I bought the Kindle in the States, sure, but can I buy books from Amazon when I’m in Finland? tells me to go to (eerily similar to my first tries of buying a Kindle). I simply don’t understand this. I get it that they need to like.. pay distributors and what not, but.. Just let me pay you for your stuff! I have the money! You have the stuff! Let’s transact!

You can also move content by sending an email to your “Kindle email address”, which was created when you first registered your Kindle. Also, you can probably use Wifi (haven’t tried it). USB is fine for me.

Even if I have to live without content from Amazon’s stores, there’s still plenty for me to read, and plenty of good publishers that provide me with cheap, compatible books.

Compatible books. What a laugh-riot.


The Trouble with Tumblr…and other stuff

So.. I’ve been thinking about two things in relation to Tumblr, the popular image-blogging site.

1) Why is it so hard to get an image in the original size? Sure, this may be theme related stuff, I’m not tumblr’d enough to say. But when you see a thumbnail on a site, shouldn’t you just be able to click it, and get the original, right? This has been the kind of.. use case since the very early days of the internet. The point of the thumbnail is to, one, decrease load times by displaying a smaller ‘preview’ image first, and letting the user decide whether he wants to load the larger image, and two, to save layout space on your site by not covering the entire screen with one image. The case in Tumblr is often that I’m clicking on an image, and then I’m taken to the comments page, where I can either click the “source” link under the image, or the link for the person (‘via xxxxx’) who reblogged the image from the original poster. Then I might get the large, original picture. Or not! I find this extremely disturbing. If  it is a theme issue, then okay, fine. But then most people are using very broken themes. It also might signify that most people have no idea how to fix the theme, or even what makes up a ‘theme’ on Tumblr. Which might, or might not, say something about the blogger on Tumblr. But enough about this angle! I digress!

2) Who provides the original content? Pick any tumblr, save for say, the official Tumblr page for a celebrity or so. Look at the images. Are they all reposts/reblogs of some other image? In some cases the reblog chain for an image is stuponfuciously long. Is there original content on Tumblr, or is like, everything a reblog of a reblog of a reblog of a reblog of some picture someone found somewhere, which was still not the original source?

Okay, I realize this is a silly thing to get annoyed over, but that’s me.. for you.

On to other things!

I’m moving. Again. I seem to live in one apartment for two to three years. But this time, it’ll be different! It’s a sweet pad. Built 2011. Four rooms, a big washroom and sauna. 98 m^2. Huge living room (I’m looking into the crystal ball and I’m seeing, yes.. a projector…). And, again, a hacking room. Same as in my last apartment. I missed that place. A room that I can fill top-to-bottom with hardware, books, whatever. A place where I can sit down, close the door and do whatever. I’m getting fuzzies just thinking about it. It’ll be great. Also, nobody will be disturbed by the humming. It’ll just be there, and it’ll be sweet.

What else what else. Didak has posted some new pics of his famous Home Office, version 7. They are the sweetness. Check them out. Waiting for a writeup or something, or a making of article. I’ve really enjoyed those in the past.

I wanted ESXi 5.5 on some Dell and HP boxes. I had no joy booting from a USB that was made using Unetbootin or Win32DiskImager. It simply wouldn’t boot. Now note, that the same image would eventually boot correctly via ILO/iDRAC using the virtual media feature. It might be a problem with the USB media I was using. Or the software that I’m using to create the bootable media. Or the specific server hardware, or the BIOS/UEFI settings of them, or UEFI in general. I googled for a solution, and I found one. Here it is! Following those instructions, I now have a proper bootable (on any machine I’ve encountered so far) media, with ESXi 5.5 on it. It might be helpful for you. Also, remember to use the vendor specific media for both HP and Dell, and not the generic VMWare Image. They contain  diagnostic tools, drivers and other stuff that will be useful later. You can find the vendor specific bootable media for HP and Dell in those two links there. These may or not be current, but they’ll take you somewhere. For Dell, google for the esxi version, and then A0x, where x is a number. When I was installing, the latest was A01.

What I’ve been reading lately: Tom Clancy’s Threat Vector (his last book?). Okay for a Clancy, and pretty eerily realistic. After that I started on Neal Stephenson’s The Diamond Age. Which has been moving a bit slowly at times. It goes from okay to excellent between chapters, so sometimes I’m reading twenty pages in one go, and sometimes it’s more like sixty or eighty. It’s a curious book, that. There are absolutely brilliant parts, and then some parts that are, to put bluntly, boring. But I’ve been meaning to read that for a while now, and I’ll be happy to finish it soon. Snow Crash was excellent, and so was Reamde. After this I will either read The Baghdad Blog, by ‘Salam Pax’, or another Clancy perhaps? I have like five books on my reading shelf.


Saying Hi to 2014

So.. 2014. Real psyched! Not really. No big plans for 2014, but there are a few milestones coming up nonetheless. My son will start elementary school this fall, which will obviously be HUGE. We’re not going abroad this year; we’re sort of planning a bigger trip in 2015 (possibly Japan), so that’s okay. It doesn’t rule out shorter trips, though. Maybe we’ll drive up north, or something.

I’m starting to get settled at my new job, and things are going smooth. Lots to do, which is not a bad thing at all. Loads of new stuff to learn, which is always great, and I need that. Not really much I can speak of, except a FreeNAS-build that I’m working on. I’ll maybe do a post or two on that build once it’s done. Sure, it’s not the EMC or IBM storage system that I often work with, but this is a cool project, that’s kind of on the sideburner. It’ll be done when it’s done.

At home, I’ve added a two-disk Iomega NAS to my network. It’s an IX-200, running in RAID1. So far, good experience. Sure, it’s not a honking big ZFS FreeNAS-box or anything, but it’ll do great as a dump/one of my backup locations. Quoting whoever said it first “If it isn’t in at least three locations, it doesn’t really exist”, refering to backups/data in general. I’ve also switched out my trusty Linksys WRT54GL to a TP-Link WR841N. It’s essentially a cheap-ish 802.11N spec access point, that can also run custom firmware. Currently on stock, but planning on putting in either OpenWRT or DD-WRT at some point. Both are installable and supported. I’ve gotten speeds of 144Mbps from the device (a laptop) to the AP, so it’s not all too bad.  The stock firmware doesn’t actually look too bad on that thing either. Gotta see if there’s something that actually is missing, which would warrant replacing it.

What else what else.. Well, I got a Sanyo Eneloop quick charger, with four AA Eneloop batteries. Everyone’s been saying good things about the Eneloop-series of products, so I have great expectations.

Updated my pf-box to OpenBSD 5.4. No biggie here.

One kind of “Wanna”-project for this year would be a ZFS-box. Let’s see if I have the funds for this. It’ll be a RAM-heavy box with both SSD’s and spinny disks. WD Red is really pleasing me (running that disk in two of my boxes currently), so that’s probably my go-to as for the rotating rust. SSDs will either be in there or not. If they are, they’ll be Intel’s most probably. My use case for an L2ARC and ZIL are speculative at best, so the only reason I’d have them is for practice and fun.

Bleh. My muse is not present, so this’ll be it.

Removing trickier VMFS-datastores

Ok, maybe tricky isn’t the right word, but at least I couldn’t find anything written on this particular issue. Maybe it’s too simple a solution even for the VMware KB, but anyway.

I was cleaning out some local datastores (Smart Array 420 and 420i controllers) and ran into an issue where I was unable to remove the VMFS datastore because of a file in use error. It didn’t give me specifics; just told me that there were file(s) in use, and/or that the datastore was busy. After a fair amount of googling I started throwing some commands at it through the ssh. There’s a vmkfstools command that can break any existing locks, and it warns you that it will do it forcibly. So I tried that, given that there was nothing on the datastore that I couldn’t afford to lose (the point, after all, was to remove it). Despite grave warnings, vmkfstools was unable to break the lock and didn’t really give me a proper reason.

Looking at the vmkernel logs (/var/log/vmkernel.log by default), I saw the same references to files being in use, but no exact reference as to what files and where. No virtual machines were running anymore, and I had deleted most everything that I could off the datastore by hand already. There was a rather specific error message relating to corruption, and googling that got me exactly diddley. The datastore had had some problems previously, some hardware had been replaced, so there were a lot of variables and things that could have affected the case.

The solution, how ever, was much simpler. ESXi (5.1 update 1), a standalone server not attached to any cluster, was shoving logfiles onto the datastore I was trying to remove. Obviously, there would be ‘file in use’-errors. D’uh. So, from the host level, I went to the Configuration tab, and from there Advanced Settings. From there, Syslog -> If it is null (and it can be null), the logs are all reset if and when you reboot the host. If there’s a path, in the style of [datastore]/path, it’ll use that instead.

So for this particular case, I set a null path, which raises a warning that logs are being stored in a non-persistent location, but it then allowed me to delete the datastore (and/or detach it first) without issue.

I was probably thrown off by the vmkernel messages about corruption, though they may have played a part in why certain files and folders couldn’t be deleted by hand using datastore browser or the command line.

After everything was done, I redirected the logs back to one of the datastores, which clears the warning (no reboot needed here, or when I set the null path earlier).

I tried to find the specific error messages but I couldn’t. I may have them somewhere so I’ll shove them in here if I find them.

Some of the commands that helped me along were:

esxcli storage filesystem list ## This lists the filesystems that the server knows about, including their UUID, label and path. These are needed for many vmkfstools commands, so it’s a good place to start

vmkfstools -B /vmfs/devices/disks/naa.unique_disk_or_partition_goes_here ## This tries to ‘forcibly’ break any existing locks to the partition that may prevent you from proceeding. Didn’t work in my case, but also didn’t tell me anything useful..

vmkfstools -V ## re-read and reload vmfs metadata information

Some of the sites and blogs that helped me along:

VMWare KB article 1009570
VMWare KB article 2004201
VMWare KB article 2032823
VMWare KB article 1009565
VMWare KB article 2011220
VMWare KB article 2004605

Thanks to everyone who wrote those.

Pi musings

So now I’ve gone and done it! I am doing something with my Pi. What I’ve done is, install nginx in a jail on it. Why? Just because I haven’t done that before. I’ll talk a bit more about what I did, and how in this post.

Why nginx? Well, the primary reason is that it’s growing in market share, and because I have very little hands-on experience of it. Also because I have this idea in my head that it’s slightly less bulky than say Apache2. Many Pi-specific pages also recommend lighthttpd, but since nginx is more prevalent on the net, I chose that.

Note! You could prepare the chroot environment beforehand. If you wish to do so, jump to the appropriate heading and then come back here. This is the order that I did things in, so if you, for some yahoo reason want to follow that, read on.

The Raspbian repositories contain a version of nginx, but it’s supposedly very old. I opted to compile from source, which seemed like a good idea after the repositories listed for a more current version didn’t work properly for the version of Raspbian / architechture of the Pi. Obviously, compiling on the Pi as a rather slow process, but this isn’t a rush order. To start off, i installed some necessary tools so I could compile from source:

sudo apt-get -y install wget build-essential libpcre3-dev libpcre++-dev zlib1g-dev libssl-dev

After this, wget the latest source package for ngingx,, and unpack this to a location of your choosing:

wget and the pgp signature: wget

Get the public key for the signer of the package (in t his case Maxim Dounin)  wget

Import it: gpg –import mdounin.key

And finally run gpg nginx-1.5.6.tar.gz.acs

You should get a message about a good signature, however, it’ll not be a trusted signature. You can’t be sure it belongs to the owner. The key would need to be signed by trusted sources, in order to establish the web of trust properly. But for now, we are content.

Then once you are all wrapped in tin foil, go prepare a pot of your favorite coffee and start compiling nginx. Change, add, remove options as needed. This is just from another howto, so you might like different locations for your logs, or include modules that are not included here:

cd nginx-$VERSION ./configure –sbin-path=/usr/sbin/nginx \ –conf-path=/etc/nginx/nginx.conf \ –pid-path=/var/run/ \ –error-log-path=/var/log/nginx/error.log \ –http-log-path=/var/log/nginx/access.log \ –with-http_ssl_module \ –without-http_proxy_module make

After this, you could potentially start nginx using /usr/sbin/nginx, but we’re not done yet.


Here, we want to do some potential damage control. The webserver is living inside its own little world, and if someone gets into that world, it’s kind of small and boring, and has no real access to the underlying OS.

We can do this either manually, or by giving the chroot directory (the new root) as a variable:

mkdir $D

After this, we need to create necessary directories inside the chroot directory for nginx to work properly.

# mkdir -p $D/etc
# mkdir -p $D/dev
# mkdir -p $D/var
# mkdir -p $D/usr
# mkdir -p $D/usr/local/nginx
# mkdir -p $D/tmp
# chmod 1777 $D/tmp
# mkdir -p $D/var/tmp
# chmod 1777 $D/var/tmp
# mkdir -p $D/lib

Note that we also give permissions to tmp and /var/tmp at this stage. Just to keep them writable by everyone just like they are in the base OS. Makes it easier for non-privileged users to write temporary files during installs or stuff needed when you are running the server.  Some instructions (like the one on Nixcraft that I relied on heavily while doing this) create a lib64 directory inside the chroot. I didn’t even have such a directory in the base Raspbian, so I followed suite inside the chroot by making a lib directory.

Next, create the following inside the chroot/dev directory, but first checking their special attributes using:

# ls -l /dev/{null,random,urandom}

You’ll get something like:

crw-rw-rw- 1 root root 1, 3 Jan  1  1970 /dev/null
crw-rw-rw- 1 root root 1, 8 Jan  1  1970 /dev/random
crw-rw-rw- 1 root root 1, 9 Jan  1  1970 /dev/urandom

Note column five. 1,3 and 1,8 and 1,9.  You need to set these attributes inside the chroot too. Do a:

# /bin/mknod -m 0666 $D/dev/null c 1 3
# /bin/mknod -m 0666 $D/dev/random c 1 8
# /bin/mknod -m 0444 $D/dev/urandom c 1 9

Next, you’ll copy all the nginx files from your base OS inside the chroot. For instance:

# /bin/cp -farv /usr/local/nginx/* $D/usr/local/nginx and

# /bin/cp – farv /etc/nginx/* $D/etc/nginx

Next a tricker part. Move all necessary libraries to run nginx to the chroot. You can find out what you need by doing a:

ldd /usr/sbin/nginx

You’ll get an output similar to:

/usr/lib/arm-linux-gnueabihf/ (0xb6f94000) => /lib/arm-linux-gnueabihf/ (0xb6f6a000) => /lib/arm-linux-gnueabihf/ (0xb6f33000) => /lib/arm-linux-gnueabihf/ (0xb6ef2000) => /usr/lib/arm-linux-gnueabihf/ (0xb6ea2000) => /usr/lib/arm-linux-gnueabihf/ (0xb6d3f000) => /lib/arm-linux-gnueabihf/ (0xb6d34000) => /lib/arm-linux-gnueabihf/ (0xb6d16000) => /lib/arm-linux-gnueabihf/ (0xb6cee000) => /lib/arm-linux-gnueabihf/ (0xb6bbf000)
/lib/ (0xb6fa1000)

All of these need to go to the corresponding locations inside the chroot. There are scripts floating around for checking what you need and copying them over; I just copied them manually because I’m a pleb.  You can always come back later; nginx and any other tools you use will tell you if you uare missing any libraries, and you can copy them later.

Copy the relevant contents of /etc to the chroot. I had problems with the users inside the chroot, but it might have been something I messed up. I was unable to run it using nobody:nogroup, and had to resort to using the uid and gid, but more on that later. If someone knows what I fucked up, and happens to read this, use the comments, thanks! But the copying I mentioned (again thanks to Nixcraft):

# cp -fv /etc/{group,prelink.cache,services,adjtime,shells,gshadow,shadow,hosts.deny,localtime,nsswitch.conf,nscd.conf,prelink.conf,protocols,hosts,passwd,,,resolv.conf,host.conf} $D/etc

And some directories (though my raspbian install didn’t have prelink.conf.d at all):

# cp -avr /etc/{,prelink.conf.d} $D/etc

We’re just about done. Kill an existing nginx’s using pkill nginx or something like killall -9 nginx to do it more violently.  Then we can run a test of nginx inside the chroot. This will tell you what is missing (libraries, files etc.), or if your config syntax is wrong:

# /usr/sbin/chroot /nginx /usr/local/nginx/sbin/nginx -t

To run it finally, remove the -t at the end. As I mentioned, at this point I had issues about a line in the nginx config file (/etc/nginx/nginx.conf), which is “user nobody;”. For the life of me  I could not get it to run using this user, even though I had it inside the chroot/etc/passwd, and group files. It just told me unknown user and so on. Changing the user also had no effect, i tried creating a fresh user, but to no avail. Finally, I ended up running nginx with:

/usr/sbin/chroot –userspec=65534:65534 /nginx /usr/sbin/nginx

Where 65534 is the uid and gid (respectively) of nobody and nogroup. Note that we are chrooting into /nginx (my chroot directory for nginx) and then from there, running /usr/sbin/nginx which is the script that starts nginx. After this, we have nginx running under the correct user and group:

nobody    4355  0.0  0.1   4984   724 ?        Ss   Oct07   0:00 nginx: master process /usr/sbin/nginx
nobody    4356  0.0  0.2   5140  1228 ?        S    Oct07   0:00 nginx: worker process

To be absolutely sure that nobody runs the “base OS” version of nginx, you can remove the directories associated, or rename the executable file under /usr/sbin (i called mine nginx_nonchroot), so I can verify that file isn’t being run. Or remove the execute bit with chmod -x /usr/sbin/nginx.

When starting nginx at boot, be sure you are doing it in the right way to ensure it’s inside the chroot:

# echo '/usr/sbin/chroot /nginx /usr/sbin/nginx' >> /etc/rc.local

To verify that your nginx is running inside the chroot, use the process id (second column when you run ps aux | grep nginx; in my example, 4355), by running:

# ls -la /proc/4355/root/

…and you’re getting the contents of the chroot root, i.e. all the directories that sit under the chroot /

drwxr-xr-x 10 root root 4096 Oct  7 19:00 .
drwxr-xr-x 24 root root 4096 Oct  6 23:24 ..
drwxr-xr-x  2 root root 4096 Oct  7 19:11 bin
drwxr-xr-x  2 root root 4096 Oct  6 23:25 dev
drwxr-xr-x  5 root root 4096 Oct  7 19:43 etc
drwxr-xr-x  3 root root 4096 Oct  6 23:36 lib
drwxr-xr-x  2 root root 4096 Oct  7 00:03 run
drwxrwxrwt  2 root root 4096 Oct  6 23:23 tmp
drwxr-xr-x  5 root root 4096 Oct  6 23:27 usr
drwxr-xr-x  5 root root 4096 Oct  7 19:51 var

You can also change the default index page so you can see that that’s the one being loaded.  In my case /nginx/usr/local/nginx/html/index.html. You can reload the chrooted nginx using:

# /usr/sbin/chroot /nginx /nginx/usr/sbin/nginx -s reload

You could now make sure nginx is listening on your pi, by using:

netstat -pantu | grep nginx

tcp        0      0    *               LISTEN      4355/nginx   

Browse to the ip assigned to your pi and see your webpage! Make sure you lock things down with iptables, and allow traffic only to ports that you want, and from addresses you want.

Infinite props to Nixcraft for this article, which helped me along the way. The main reason I wrote this was that my install  was slightly different, and I figure I’d type my own problems and solutions down. Also, raspbian has changed slightly (i guess?); So here you are. This howto was also very helpful, thanks to




LSI Updates and Pi

There’s no possible way to make a Raspberry Pi-joke that hasn’t already been made.


So far so good. Things’ve been working fine, though I have to look into disabling the bios since I’m not booting from any drives that are behind the LSI card. Boot times are three times as long as without the card, even though the OS is loading from the Samsung 840 Pro SSD drive.

I used MegaRaid Storage Manager for Windows to install the latest BIOS for my card. I went to the LSI site, searched for Host Bus Adapters -> LSI SAS 9211-8i -> Firmware, and downloaded the only available package (at the time this was named “9211-8i_Package_P17_IR_IT_Firmware_BIOS_for_MSDOS_Windows”, released Aug 09, 2013, the same package as for the IR-firmware installed in the previous post). Inside the archive, you will find various folders. Look in the  folder “sasbios_rel” and check that you have mptsas2.rom in there. That’s the BIOS image.

The good news is, as I mentioned, once you have the Storage Manager software installed, and your card is recognized, you can flash the BIOS from Windows without issues. This should also work for Firmware, but I haven’t tried this yet, as I am already running the latest IR-firmware. Open up SM, and somewhere in the middle you will find Update Firmware. There, select BIOS (middle selection for me), and browse to the folder mentioned earlier. Inside, select the mptsas2.rom file. Hit OK, and it will ask you to check a box and confirm that you want to update the BIOS. After that, it’ll flash, and tell you when it is done. It will show you the old BIOS version until you reboot. My card was, and is now Improvements are minimal, but there were some.

One note on the Write Cache, mentioned in the last post. I was unable to enable this from Storage Manager. Perhaps due to the fact that there is no battery backup unit. I’ll have to look more into this at a later date.


Got me a Pi. The B model, from local RS reseller, Yleiselektroniikka. Cost me 47 bucks including taxes. It’s the revised Model B, with 512MB memory. I also got a transparent case, which was 10 bucks. I didn’t get a powersupply, because I have plenty of USB chargers for various devices (and a few generic ones) that provide 1A+ @5V. My HTC Desire Z charger powered the Pi just fine, even though there’ve been reports of “flaky” mobile phone chargers not working with the Pi.

I have an 8 GB Verbatim SD-card for this project, and I dropped the latest NOOBS image from the Raspberry Pi homepage on the card, after formating the card FAT. I then installed Raspbian from the NOOBS-installer, and proceeded to do an apt-get update && apt-get upgrade, which also upgraded the Pi bootloader to the latest version (as was recommended by the small booklet that came with the Pi.)

I haven’t done much with the device yet (joining the club of Pi owners everywhere! :)), except hook things up and tried it out a bit. It works great! Or just as advertised. Obviously the boot is a little bit slow, but nothing out of the ordinary, considering the specs. HDMI out works fine; I use an HDMI -> DVI cable for this.