Quickie - Enable remote powershell without touching the PC

Quick and convenient way to get WinRM up and running without having to touch the PC.  Useful if you prefer to leave it disabled in your environment unless you need it or have decided to deploy it to a number of existing PCs.  Use scheduled tasks to enable it for you (note: this isn't my trick, so if you're looking for this you probably already saw it elsewhere, just sans commands!)

PS C:\> schtasks.exe /Create /S ComputerName /TR "powershell -ExecutionPolicy Bypass -Command Enable-PSRemoting -Force" /TN EnablePSRemote /SC MONTHLY /RU "SYSTEM"
SUCCESS: The scheduled task "EnablePSRemote" has successfully been created.

PS C:\> schtasks.exe /Run /S ComputerName /I /TN EnablePSRemote
SUCCESS: Attempted to run the scheduled task "EnablePSRemote".

PS C:\> schtasks.exe /Delete /S ComputerName /TN "EnablePSRemote" /F
SUCCESS: The scheduled task "EnablePSRemote" was successfully deleted.

PS C:\> Enter-PSSession -ComputerName ComputerName
[ComputerName]: PS C:\Users\account\Documents>

App-V 5 Release Impressions - Part 2

Now that I've had some time with App-V 5, I have a bit more feedback.  There's been a number of quality posts on the App-V blog lately to provide a little more insight in to the changes.  No matter how you look at it, Connection Groups are vexing beasts.

Here's the thing - I love the idea of Connection Groups and I really want it to work.  The multi-component installs of desktop software that are so often the things nightmares are made of (especially in a heavily outdated Oracle shop) would be made considerably more convenient if you could maintain the components separately.  I'm sure Citrix has an implementation of the same concept that's totally fantastic too.  So far, however, it's just been a bit of a hassle.

It works, don't get me wrong - I have Office 2010 sequenced and two different Outlook plugins in separate packages all in one Connection Group and they work grand.  Didn't take any effort at all.  However, at one point I had forgotten to enable local COM interaction or something in one of the packages and published it and saw what a mess you can make if you don't thoroughly test (something I never do when I'm learning, I want the messes to show up).  It's not super obvious what will and won't work though - if I put a separate Visio 2010 package in the same Connection Group, all of the applications cease to work.  If I leave it separate, everything works dandy (including cut and paste of Visio objects to Word/Outlook/etc - well, it spits out a weird file save error before it pastes, but it works better than it did in App-V 4).

Other than the unpredictability of Connection Groups I've been very pleased.  I'm running a Win7 x64 desktop with all virtualized apps now (with the exception of anything that has an IE plugin) and it works great.  The server infrastructure has been very reliable and easy to teach to others.  Startup times are the same or faster than locally installed apps.  The powershell interface is super useful (especially with remote ps enabled on the desktops).  Troubleshooting is considerably easier than it was.

By the way, I may have mentioned this previously but if you're doing cacheless streaming, turn on whatever compression mechanisms your web server employs.  Trust me.  My package servers are behind BigIPs.  Without compression, it takes Outlook 30-40s to start every time.  With compression on, 3-5s.  Gigantic difference.

I've just released our first App-V 5 + Win7 x64 SCCM build for courageous testers/volunteers in the company who need new systems, so I expect as the usage expands I'll find more problems and I'll be sure to report them here.

App-V 5 Release Impressions - Part 1

It seems that it's not the most popular kid on the block, but I'm a sucker for App-V.  We're a 50/50 shop (virtualized apps/installed apps) currently and it's been an excellent tool for us.

I've been looking forward to App-V 5 for a while and read through the various beta impressions but they've mostly been either very technical how-tos on things that may not be (strictly speaking) supported or a bland two paragraphs on the fact that it's 'different'.  So, I thought I'd throw together some stuff here that might help fill things out for people.

Server Side - Install
I genuinely don't remember installing App-V 4.x server side, but I think it was vaguely uneventful.  The App-V 5 server install is similarly uneventful as long as you're prepared.  Pre-reqs that you likely won't have installed already but will need are .NET 4.x (4.5 is the Features option in Win2012 which seems to work fine), Powershell 3.0 (aka WMF 3.0 / KB2506143) and some random insecure library loading vuln patch (KB2533623).  Roles and features you're primarily looking at IIS stuff, nothing too surprising - I think ASP.NET with registration to .NET 4.x is the only thing to watch out for.  On Win2012 I didn't have to do any hoop jumping.

Server side is split in to multiple parts:
- Publishing, Management and Reporting Servers (each installable separately)
- Management and Reporting Server Databases (each installable separately, must be on SQL Standard or greater, no Express)

Installation is relatively straight forward, I installed the two DB roles on a SQL 2008R2 server and the three server roles on their own server (each utilizing a different IIS port).  Did I mention the entirety of the management and configuration is really done via Powershell now?  There's a web based console for the management server that'll make you wish you had spent more time using Powershell, but you're still welcome to use it.  I think the only functions that are available in it are setting App-V admins and importing packages.

App-V 5 Management Server Web "Interface"

Client Side - Install
Same prereqs as the server side - .NET 4.x, WMF 3.0, and that one-off vuln patch.  Other than that there's literally no configuration - I think the expectation is that you'll control all of your clients via GPO or (I'm guessing these are available) command line switches.  Post-install all configuration options are only available via Powershell.  Thought I'd toss in a screen of this thing too since I'd never seen one prior to install.  I like the idea that it's much simplified and user friendlier.

App-V 5 Client GUI

Sequencer - Install
Also the same prereqs as the server side.  You'll notice that the sequencer didn't get the big face lift that the other two components got, so you'll feel right at home if you sequenced in App-V 4.6.  The Q: recommendation is gone as is RTSP (yay!) so for me the sequencing is much simplified when teaching it to other people (when folks are new to packaging in general adding in the weirdness of capturing to Q:\[8.3] and then post capture having to enter in the client-accessible package URL was always a nuisance to me).

More on this later - I've sequenced a couple quickie apps and setup the publishing behind a hardware load balancer and some other stuff that someone might be interested in.  So far everything is working stellar, so I'll report back as I progress.

Testing a 32-bit .udl on a 64-bit Windows OS

Really just mirroring this blog post in case it's really gone for good (google cache still has it thankfully).

If you need to test a 32-bit ODBC datasource on a 64-bit version of Windows _and_ there's no native test function (i.e. you're using the Microsoft ODBC for Oracle driver which is 32-bit only and has no test button), you can still use a .udl file to assign the provider and datasource to test.  It just takes a little extra legwork.  Run this and all will be well:

C:\WINDOWS\SYSWOW64\rundll32.exe "C:\Program Files (x86)\Common Files\System\Ole DB\oledb32.dll",OpenDSLFile C:\path\to\test.udl

Quick F5 BigIP Note

In case you're wondering (and I'm sure if you work with F5s often enough you've already figured this out), if you ever need to switch your Self IPs to different physical ports/VLANs and you have Floating IPs configured, just delete the Floating IPs, change to the new VLAN/interface/whatever, then add the Floating IPs back.  It's just easier than any cockamamie scheme you might be trying to concoct to avoid doing that.

Fun fact about OpenVMS 7.3-2 multipathing

Discovered this week that OpenVMS 7.3-2 apparently doesn't turn on its multipathing brain unless it has a second HBA installed, in Fabric mode (via wwidmgr), and zoned over both HBAs.  I imagine it's just a side effect of the multipathing logic pre-dating SANs with multiple paths to a single HBA, but it sure caused quite a bit of consternation when I was prepping to do SP failure testing on an EMC VNX and couldn't get paths to show.

Random AIX 5.3 Notes - expanding a logical volume

I've spent the past few days working on getting a PowerPath issue resolved on an AIX 5.3 box.  I rarely touch these things so my familiarity is abysmal making this much more difficult than it probably needs to be.  So I thought I'd just scribble down a few random things that I've worked through in hopes that it'll help someone else.


Expanding a logical volume without adding disks for those of us still stuck in the 90s (assumes it has already been expanded at the SAN level):


# Unmount, varyoff and export the volume group
bash-2.05b# umount /mountpoint
bash-2.05b# cfgmgr
bash-2.05b# varyoffvg thisvg
bash-2.05b# exportvg thisvg


# Deal with pv weirdness
bash-2.05b# chdev -l hdiskdevice# -a pv=clear
hdiskdevice# changed
bash-2.05b# chdev -l hdiskdevice# -a pv=yes
hdiskdevice# changed


# Recreate the volume group | -Y NA means don’t automatically rename the logical volumes
bash-2.05b# recreatevg -Y NA -y thisvg hdiskdevice#
0516-1434 varyonvg: Following physical volumes appear to be grown in size.
        Run chvg command to activate the new space.
        hdiskdevice#
thisvg
0516-1434 varyonvg: Following physical volumes appear to be grown in size.
        Run chvg command to activate the new space.
        hdiskdevice#


# check and grow vg - hello free PPs!
bash-2.05b# chvg -g thisvg
0516-1164 chvg: Volume group thisvg changed.  With given characteristics thisvg
        can include upto 7 physical volumes with 4064 physical partitions each.
bash-2.05b# lsvg thisvg | grep "FREE PPs" 
MAX LVs:            256                      FREE PPs:       856 (54784 megabytes)


# Take care of the filesystem - /fs is automatically added to the mountpoint when it’s recreated
bash-2.05b# chfs -m /mountpoint /fs/mountpoint
bash-2.05b# mount /mountpoint


# if needed, increase allowed LPs in LV
bash-2.05b# chlv -x 4686 thislv 


# Grow the filesystem, confirm the change
bash-2.05b# chfs -a size=+54784M /mountpoint
Filesystem size changed to 419299328
bash-2.05b# df -k /mountpoint
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/thislv   209649664  74832836   65%      228     1% /mountpoint

App-V - Running apps in compatibility mode

Fun fact - you can add App-V virtual drive paths to the compatibility mode registry key and have your virtualized apps run in compatibility mode.  Normally, the shortcuts for App-V apps all point to sfttray.exe, so if you try to set compatibility on the generated App-V shortcut, you'll actually be setting it on sfttray.exe for all App-V apps.

Just head over to HKLM/Software/Microsoft/Windows NT/CurrentVersion/AppCompatFlags/Layers.  Create a new REG_SZ where the name of the item is the virtualized path (for example, Q:\MyApp.000\MyApp.exe and the value is the compatibility mode (WINXPSP3 for example).

The downside of this (at least so far) is that when I set this to WinXP something trips in the App-V stack to now require admin privs.  If I could make that magically go away, this would be a good day.

Dell w/ ESXi BIOS updates - easy as pie

It's such a lame topic.

Having trouble just getting a functional disc or whatever together to update the BIOS on a Dell server running ESXi?  Use this:

http://linux.dell.com/files/openmanage-contributions/omsa-64-live/OMSA64-CentOS55-x86_64-LiveCD.iso

And put a copy of the BIOS that you need (from the Linux drivers section of support.dell.com - .bin file) on another server on the same network or pop it on to a USB stick (or even another CD/DVD).  Copy the BIOS to ~, chmod 777 the file and execute it.  Easy as pie.  I can't believe this very simple method wasn't plastered all over the internet in place of the 4 bajillion websites that suggest doing a myriad of other painful things.  Maybe I suck at searching, who knows.

ADM template to disable password saving on IE/Windows authentication dialogs

I don't like ADM templates. Surely there was a better way to implement this thing (I haven't looked at an ADMX yet, but I suspect it's even worse).

This is an ADM template to prevent people from saving their credentials in those proxy pop-up dialog type boxes that you tend to come across for things like IIS/Apache auth or outbound proxies (like internet authentication). If you ended up here you were probably looking for this, so here it is:




  CLASS USER 
  CATEGORY "Internet Explorer Password Caching" 
    POLICY "Disable password caching in Internet Explorer" 
      KEYNAME "Software\Microsoft\Windows\CurrentVersion\Internet Settings" 
        VALUENAME "DisablePasswordCaching" 
          VALUEON NUMERIC "1" VALUEOFF NUMERIC "0" 
    END POLICY 
  END CATEGORY; 

The important things to note are:

1. In the GPO editor, highlight Administrative Templates and from the top menu choose View -> Filtering... and uncheck "Only show policy settings that can be fully managed." Otherwise you won't see it.

2. Due to the nature of this key, I expect a very crafty user could open up regedit, change the key to 0 (since it is in HKCU) and save their credentials prior to the GPO refreshing it back to 1. I haven't tried it, but if you have users that would have the wherewithal to do that, I'd think silly little blockades like this wouldn't be enough anyways.  Also, why aren't you blocking regedit.exe and friends from being run?  HrrrmmMMM?

Deleting a Nexus 1000V from vCenter

Pro-tip: If you're planning on deleting your Nexus 1000V dvs from vCenter, hang on to that VM just a little while longer to run no vmware dvs if you can.  Otherwise you have to do all this stuff. The directions on VMWare's site are spot on, just a lot of wasted time that's easily avoided. If you're reading this though it's probably too late anyway!

Oh, it goes in the "repository" directory

File under: Dell Repository Manager and SUU / Dell Systems Build and Update Utility.

If you're making a USB stick with firmware / etc updates, just export it to your local disk, go to the directory it exported to (it'll be named something like SUU_2012-05-04_07-21-33-AM), swipe the "repository" directory out of there and put it on the root of the USB stick.  I don't know why this isn't better outlined in the documentation, but maybe I'm not good at reading things.

Emulex OCe10102-FX firmware upgrades

If for some reason you ever find yourself in the undoubtedly very common situation where you need to update a very old firmware/boot code version on some Emulex 10g OCe10102-FX cards that you just received and they're installed in an ESXi box, have I got some delightful news for you!

Maybe not delightful.  But, since I couldn't find any references to this anywhere, I thought I'd share.

Here are what I considered my options to accomplish the task at hand:

Option 1: So there's this Emulex OneConnect vCenter plugin that's a bit weird but I expect gets the job done in better cirumstances - it actually requires (for lack of a better term) a proxy Windows server that runs a service that does all the heavy lifting (I just ended up installing it on my vCenter server for my own sanity).  Fortunately, it has an interface to update the firmware.  Unfortunately, attempting to update the firmware via that interface just resulted in a "job" that sat out there forever and did nothing until I restarted the (Windows) service.

Option 2: There are some fine instructions here that give a very straightforward look in to creating a WinPE boot disk with the Emulex offline utilities and Windows AIK that can be used for updating firmware.  However, when I went to use my disk, my cards were all identified by the Emulex drivers as OCe10100 cards and they wouldn't update the firmware due to it being for a different adapter type.

Option 3: A loatheful option, but the one I started chasing - build up a Windows or Linux server and plug as many cards in to it as possible and just bulk update them using the normal Emulex tools.  This isn't terribly helpful if you don't have a spare server lying around or if you're, I don't know, in any sort of hurry, but I couldn't figure out any other way of pulling this off.

On a whim I shot an email over to Emulex's technical support telling them about the misidentified cards in a WinPE environment and they responded with this:

This is an indication that the firmware on the adapter is fairly old. To work around is a bootable ISO image that will install the newer firmware version into the adapter.

Due to release policy inside Emulex, I will need to request the ISO be sent to you. You will receive an email from another Emulex employee to get the ISO.

Oh, I get it.  My firmware is too old to update to a newer version.

In any event, no ISO yet and my Windows box will be plugging away at the cards in the morning if it's able, but there it is.  If anyone's curious, the current firmware on most of these cards is 2.703.269.30.  Seeing as the current version I downloaded is 4.0.360.3, I suppose I can see how issues might arise.

Edit: The provided ISO from Emulex worked dandily!

One Liner to Patch an Offline Win7 Image

Keeping the base image for desktop deployments up-to-date with patches is important, if anything to avoid the inevitable super long patch session when it either first comes online or hits the first patch distribution cycle of its life.  The other day I was just trying to figure out if it was possible to do it in one line - not because that's necessarily useful (it's not), but mostly because I just like goofy stuff like this.  This assumes that you have cab files downloaded via WSUS or some other method and dropped in a directory and that your image is already mounted to F:\WIMMount.  Change the "x86" to "x64" if you're patching a 64-bit image.

C:\Program Files\Windows AIK\Tools\x86>forfiles /s /p F:\WSUS\UpdateServicesPackages\ /m *x86*.cab /c "cmd /c dism.exe /image:F:\WIMMount\ /Add-Package /Packagepath:@file"

Active Directory Security Event Log Monitoring with Powershell + Nagios

For the longest time I've been using a strange combination of snmptt, a duct tape perl script, and Nagios to monitor basic security events in a Windows 2003 domain - basically, the DCs would push SNMP events for failed logins and locked accounts to snmptt and the Nagios check would run the script that parsed the snmptt log over a period of time to determine whether or not there have been too many failed logins or locked accounts (indicating an account problem or a brute force attempt). 

I've always hated this, but I think as a rule I always hate the first way I do something because it always looks entirely ridiculous in hindsight.

So, while in the process of migrating Nagios to a different server, I decided it was time to cast off the shackles of SNMP and do this a little more sanely (if redoing it in Powershell can be considered sane).  A co-worker of mine has been lauding Powershell as the second coming for quite a while so I decided to dive in and really it ain't bad. 

For example, just to get the events I wanted during the timeperiod I wanted (i.e. in the last hour):

$events = get-eventlog -logname security -EntryType FailureAudit -After (Get-Date).AddMinutes(-60)

I learned a few little oddities about security logging while doing this - for example, EventID 675 (Pre-authentication Failure) with a Failure Code of 0x19 is an artifact of having newer client machines (in this instance, Win7) attempting an encryption method not compatible with a 2003 domain - the system will try a compatible method immediately afterwards.  So, those can be safely ignored by adding an if statement / whatever that strips those out (note: all the meaty parts of the event are stored in the replacementStrings array of the event object - unfortunately, since each EventID stores different info in the message there is no rhyme or reason to what data will be where in the array and you have to set up some per EventID filtering to get the data you want):

$event.replacementStrings[4] -ne "0x19"

But wait!  EventID 672 (Kerberos authentication ticket request) with a Result Code of 0xE appears to be a similarly ignorable issue but from a different source - looks like it might be coming from ActiveSync in my case.  So, the if statement can be updated (of course the 672 events put the Result Code in a different part of the replacementStrings than 675...):

$event.replacementStrings[4] -ne "0x19" -and $event.replacementStrings[6] -ne "0xE"

So if you've been messing around with this to see how it looks, depending on where you're running it you may have noticed an inordinately long pause after it returns the results - in my case, the pause is about 60-90s (and this only occurs on my DCs).  I have no idea why that is - my guess is that it checks at least some of the other events in the log to see if their timestamp is prior to the time listed in the -After switch (thus negating some of the usefulness of the switch).  If anyone has a more solid answer to that I'd love to hear it.

Once you have the data you want out of the event log it's pretty easy to get it back to Nagios - like any other script as long as you write a line of text and Exit with the right return code you've got a good chance of success.  I've recently switch to NSClient++ after years and years of using NRPE on Windows and I must say I much prefer it (and it's still actively developed which doesn't hurt at all).  Using the premade config option for wrapped Powershell scripts (ps1=cmd /c echo scripts\%SCRIPT% %ARGS%; exit($lastexitcode) | powershell.exe -command -) I just added it to the scripts directory and added a directive in wrapped scripts and it works dandily (note that if you try to do a more common syntax to call Powershell like powershell.exe -File script.ps1 NSClient++ will just go out to lunch, so don't try to get fancy).

Voila!

# ./check_nrpe -H domaincontroller -c check_failed_logins -t 120
OK: 55 events generated since 7:51 AM

I'll add the whole script to http://exchange.nagios.org/ later and link it here for anyone that's interested.

Brief Introduction

I recently read a blog where someone had a philosophy of blogging anything that took them longer than an hour to fix.  I've decided that I'm going to co-opt that philosophy.  Lately I've been stumbling across a lot of odd problems in my day to day endeavors as a sysadmin, so what better way to give back to the internet that I syphon off of daily than to (hopefully) help someone in the future.

I spend most of my days managing a little bit of everything - Windows, Linux, Unix, OpenVMS, SANs/fabrics, VMWare - so there may be little to no consistency in what's posted here.