Value of EA on IT projects paper published!

I’m happy to announce that my paper “Measuring the value of Enterprise Architecture on IT projects with CHAOS Research” has been published in the Journal on Systemics, Cybernetics and Informatics (JSCI) Volume 15 – Number 7 – Year 2017, pp. 13-18 ! This paper summarizes the research I’ve done together with Jim Johnson of The Standish Group for my master’s thesis for the program of Enterprise IT Architecture at the Antwerp Management School.

You can read the whole paper online here: http://www.iiisci.org/journal/sci/FullText.asp?var=&id=JS229KF03. It’s only 6 pages, with more than 3 pages full of graphs, so it’s a rather quick read.

TL;DR: The value of our research is that we’ve found empirical evidence that IT projects are more successful when organizations have an EA.


A remark I hear often though is “The value of EA seems rather small with these results [and contradicts what’s commonly said about the value of EA]”. Let’s explore that a bit:

Firstly we’ve not looked into the value of EA as a whole. We’ve looked into the value of EA on IT projects. We had to scope the research, otherwise it would be too broad & difficult to measure, as EA provides benefits on a wide range of domains and levels. So this is just a subset of the value of EA.

Secondly, research done by The Standish Group has already proven that the success of IT projects is mostly influenced by

  • the project size,
  • project methodology,
  • experience of the project sponsor,
  • emotional maturity of the organization,
  • and the skill level of the delivery team.

To put things into perspective; if, hypothetically, projects would be 100% successful after the introduction of EA, this would mean that EA would solely be responsible for the success of an IT project (and the previously mentioned succes indicators would not). It would be absurd to even think this would be possible, certainly if you know that EA has the most effect on the design phase of a project, not the implementation phase. In that perspective, the average increase of 14% of successful projects (when measured by timeliness, budget adherence & satisfaction) is rather as expected. Certainly if you know that common waterfall project methodologies as PRINCE2 have a detrimental effect to project success. If you want to know more about the general success factors for IT projects, I recommend to order the latest CHAOS report of The Standish Group.

Thirdly, we have not researched what the client meant when saying that the organization has an EA. For example, we did not ask if they were using an EA framework (and hence also didn’t ask which). It could be that a certain EA approach/methodology has more success then othersWe also did not measure the form/maturity/quality of the EA practice. These would be interesting questions to ask in follow-up research, as these would also answer the age-old questions “which EA framework has the most value” & “how to best implement/approach EA”.

Fourthly, the value of EA on IT projects increases as the size of the project grows (with projects with a budget of $10 million or more as an exception to this “rule”). As the project size increases, the budget also increases, which means that the increase of success by EA has an even greater financial benefit.

I think the most important conclusion and takeaway is that we finally have empirical evidence for the value of EA, albeit only scoped to the value for IT projects.

If you have other remarks or questions regarding our research, I’ll gladly answer them – just leave your comments below.

Email continuity? Are we really discussing this?

OK, I’ll bite. This topic came up recently after an outage of Azure and Office 365. Because of this outage countless companies weren’t able to send and receive email for a couple of hours. Companies like Barracuda, Intermedia and many others are jumping on this to sell ‘email continuity’ solutions. They promise the availability of email even when the email provider goes down. So there must be value in these solutions…. or not?

Let’s take email: the advantages of email over physical mail are notable; it costs much less, is more reliable and delivery is much faster. Just those three are responsible for the decline of philately as a hobby.

However, email is also a very misused medium. Especially because people have gotten used to its shorter delivery time and good reliability, it’s now even being used for time-critical business processes. Two things that email’s not; 1) completely reliable and 2) real-time.

Email can get stuck. For example, if a spam filter along the way goes overzealous and marks your email as being spam, it’ll likely not arrive – or in the event it does, it might arrive in a folder that doesn’t get checked.

Email can arrive late. For some reason (e.g. because of a full mailbox) mail can just hang in the outbox and doesn’t get sent. And if you don’t have an internet connection you won’t be able to send or receive anything.

One of my customers was interested in ‘email continuity’ because they take orders in via email, which are then automatically processed and hence they consider this process as business critical. I won’t deny that sales is the lifeblood of companies, and I won’t say either that mail is a bad medium for taking orders. What I am saying that using email for time-critical processes is wrong. So the real question is; is taking in orders time-critical? As always, this depends 🙂

Let’s take Amazon.com. Being able to ship products incredibly fast is a competitive advantage compared to other online and offline resellers. The order process for Amazon is thus considered time-critical.

Just fantasize for a moment that Amazon would be taking in orders via email. Millions of emails each day flowing into a shared mailbox. In such a case an ‘email continuity’ solution would make sense because outage would mean lost sales to a competitor.

However, it’s quite obvious they wouldn’t be able to guarantee their next (or same) day deliveries because of the aforementioned email issues, let alone the management of that mailbox. You could automate the mailbox, but automating unstructured mail is immensely challenging and in a lot of cases undoable with current technology. It just wouldn’t work.

Now take a consultancy company which receives RFP’s or offer requests for large projects, where the expected time for a response is 30 days or more. In such a case it doesn’t actually matter if such a mail is opened the same or next day. Here it can make sense to use email, and the outage of an email provider would just be a minor issue. Still, using email, there’s the matter of email not guaranteeing delivery, so you must accept this (albeit low) risk.

So in both cases email continuity isn’t necessary. In the first because the process is time critical and email can’t be used to achieve the desired performance, timeliness and reliability. In the second because the process isn’t very time critical, and next day delivery of email is considered acceptable.

In conclusion: if you fear email outages and want an email continuity solution, it’s usually masking another problem. Most likely one or more business processes would be better off with something else than email. So are you going to alleviate the symptom or fix the real issue?

HTTP to HTTPS redirection

You’ve encountered it before probably. You want to securely publish a webapplication. The webapplication works with the http protocol and thus hosted on port 80 (typically) on your webserver. One of the requirements when publishing externally is to this securely which means encrypting the http traffic with TLS (https).

You go ahead and use your reverse proxy to open up an externally available https port (typically 443), install a certificate and point the traffic to your webserver on port 80. This means that the client hits the reverse proxy on port 443 (so this traffic is encrypted), but internally (from the reverse proxy to the webserver) the traffic is unencrypted. So far, so good.

However, when you test the webapplication you get the dreaded “page cannot be displayed” (or something of sorts). You start troubleshooting. You can ping and you can successfully access port 443. You start monitoring your webrequest with a tool (e.g. Fiddler, Wireshark,…) and notice that your request does get a response from the webserver, but the subsequent request fails. Then you see the problem, that subsequent request is not hitting port 443, but port 80! Then you look at the response of the webserver. And there’s the culprit, you see a 302 redirect to http://…

This means that the webserver is successfully receiving the request, and sends a response back, however the response does not consider that the client needs to hit the reverse proxy on port 443. Indeed, the application is unaware of this (it gets http requests from the reverse proxy) and just sends http responses (with http:// urls in the body of the response) back.

Here I’m going to present several solutions for this problem.

Solution 1

The most preferred solution is to make the web application aware that it’s being fronted by a reverse proxy. Some applications are aware of these kind of configurations and can be configured for this (SharePoint is a prime example). Other applications just need a hint. This involves adding a http header (X-Forwarded-Proto: https) in the request on the reverse proxy. This particular header helps the webapplication to identify the protocol (https in this case) that a client used to connect. Knowing this, the webapplication then serves https urls in the body of the response, while still sending the response with http.

Advantages

  • Easy to implement
  • No or minor changes to the reverse proxy
  • No or minor changes to the webapplication

Disadvantages

  • The webapplication must support this

Solution 2

You could listen also on port 80 (http) on your reverse proxy and redirect those requests to 443 (https)

Advantages

  • No changes required on the webapplication
  • Minor change on the reverse proxy

Disadvantages

  • Introduces a very small delay on the client (1 extra response + request) per hit on port 80, could become noticeable if page contains alot of http:// links
  • Some types of content (e.g. HTML5 streaming video, streaming in general) will not like this
  • Less secure. Redirecting http to https is prone to Man in the middle (MITM) attacks (thanks to Jan Tytgat for this suggestion)

Solution 3

Rewrite the body of the web page on the response to modify http:// urls to https://

Advantages

  • No changes required on the webapplication

Disadvantages

  • Introduces a slight delay on the reverse proxy (when modifying the body), normally only noticeable with millions of hits per second though
  • Difficult to configure. You need to configure the policy to only change links to your webapplication. If you change ALL urls with mixed content (for example images on another webserver), you would also change the urls to external content which might not be accessible on https. Even more difficult when client-side javascript is being used where urls are being generated in code.

Solution 4

Forego http completely, and configure https on the webapplication.

Advantages

  • More secure
  • Minor change on the reverse proxy (address the webserver on port 443 instead of 80)

Disadvantages

  • You need to configure a certificate on the webserver too (two places to address when updating a certificate)
  • Needs support from the webapplication (will need to be configured for https)
  • Webserver might already have another application configured on port 443. This can be resolved by using host names or using another port (e.g. 444), however the latter will introduce accessibility issues when the application is also accessed internally (the user will need to explicitly add the port to the url)
  • Additional performance hit on webserver (must now also process encryption)

So there you have it, 4 possible solutions, with the first one preferred, and 3 others if your webapp doesn’t have out-of-the-box support.

SSO to SharePoint and XenApp with NetScaler, ADFS and Kerberos

Isn’t the holy grail of user experience not the possibility to only logon once and never to enter credentials again? We can do that today, and with the NetScaler 11 release this is even easier than ever.

In this post we’ll be chaining products like ADFS, SharePoint, Citrix XenApp and NetScaler with technologies like SAML and Kerberos. We’ll also be using the new Unified Gateway functionality to do everything with a single external IP. I’m also assuming some basic knowledge of ADFS, IIS and NetScaler Gateway. Indeed, this is not for the faint hearted.

So what is the use case? Well, one of my customers wanted a solution to provide their users with Single Sign On (SSO) access to all of their applications. This solution would be composed of a portal site containing links to all applications (web apps, or Windows apps hosted on XenApp), and an access point consolidating various authentication methods and providing federated access. The customer has internal users, but also has employees and associates working at universities spread all over the country, who would like to login with their university account instead of an AD account of the customer.

The solution should reduce the number of helpdesk calls to reset passwords – especially for users coming from a federated partner, but also calls for getting access to applications.

Design

design

This specfic solution requires an Active Directory account for each user in the resource domain to access XenApp applications. Federated users thus also have an AD account, which we’ll lookup using the email address of the user.

Do note that this reliance on Unified Gateway requires the NetScaler Enterprise Edition. But then again, when you’re using the NetScaler to authenticate your users to web applications you should already have the Enterprise Edition for AAA functionality.

Configuration

So without further ado, I present the solution:

  1. Configure ADFS
  2. Configure NetScaler for ADFS authentication via SAML
  3. Configure NetScaler for Kerberos Contrained Delegation
  4. Configure NetScaler Unified Gateway
  5. Configure StoreFront for Kerberos Contrained Delegation
  6. Configure the XML Broker to accept Kerberos tickets
  7. Configure web resources to accept Kerberos tickets

Configure ADFS

First we’ll add the NetScaler as a relying partner for ADFS. To do this, you can generally follow CTX133919 – How to Configure NetScaler SAML to Work with Microsoft AD FS 2.0 IDP.

Now we’re going to make sure that we get the User Principal Name for each account that’s logging in (for both users in the resource domain as Federated users). We assume that the identity provider for the Federated users delivers his/her email address as a claim.

For these Federated users we’re going to lookup the User Principal Name using the email address of the user. We do this by adding an Issuance Transform rule of the type “Send claims using a custom rule”:

ADFS - Relying partner claim rule - Get UPN

In this example the email address is of the claim type “http://contoso.com/partneremail”. We use the value of this claim to lookup the User Principal Name of the user.

Note that this rule only executes when the claim of this type is given, for users authenticating with their resource AD credentials, this rule is not hit.

Then we’re going to transfer the User Principal Name As the Name ID claim by adding another Issuance Transform rule of the type “Transform an Incoming Claim”:

ADFS - Relying partner claim rule - Transform UPN to Name ID

This rule is hit for all users. As you will see in the NetScaler configuration later on, we will use this Name ID claim.

Configure NetScaler for ADFS authentication via SAML

Before configuring ADFS authentication we must install at least one (but two recommended) certificate.

  1. we need the server certificate of the ADFS server
  2. we need a certificate to sign our SAML traffic (for this we need the private key). Because this certificate is only used in traffic between the NetScaler and ADFS server, this can be a self-signed certificate.

So if you haven’t installed these yet on the NetScaler, do so now.

NetScaler - Configure Authentication SAML ServerIn the NetScaler, head to your Authentication Polices and add a new SAML server.

For the IDP Certificate Name, use the server certificate of the ADFS server.

For the redirect URL use https://<ADFS FQDN>/adfs/ls/formssignin.aspx

For the User Field, enter “Name ID”

For the signing certificate, use the certificate that you prepared for SAML signing

For the issuer name, you can use any arbitrary name, as long as you configure the same identifier name in ADFS. In this example I use the URL of the Unified Gateway.

Reject Unsigned Assertion you may leave on (should pose no problems if you’ve configured the certificates correctly).

For the SAML Binding, choose POST Binding.

 

After you’ve created the Server object, you can create a SAML auth policy. As the expression, you can use “ns_true”, as the action you select the Server object you’ve just created (srv_spadfs in this example).

We’ll use the SAML auth policy later when configuring the NS Gateway instance of the Universal Gateway.

Configure NetScaler for Kerberos Contrained Delegation

To authenticate to web servers, we’ll be using Kerberos. Because we’re only getting the UPN of the user (not the password), we’re going to depend on Kerberos Constrained Delegation. This allows the NetScaler to authenticate on behalf of the user to other servers which accept Kerberos tickets.

There are several ways to configure access to authenticate and request Kerberos tickets on NetScaler (using a keytab file, using certificates or with a user account). Here we’ll be using a user account.

First, create a user account in Active Directory to which we’ll delegate rights to access other services. Best is to create it as a service account (do not expire the password).

NetScaler - KCD AccountThen configure the KCD Account on the NetScaler. Under Security > AAA > KCD Accounts, add a new KCD account:

Give the account a new name (e.g. kcdaccount1). This name is only used on the NetScaler.

For the Realm, enter the Kerberos realm of the resource domain (e.g. CONTOSO.LOCAL).

For the User Realm, enter the Kerberos realm where the users reside. Because in our case all users have a user account in the resource domain, this is the same (e.g. CONTOSO.LOCAL).

For the Delegated User, enter the exact name of the AD service account you’ve previously created, and enter the password as well.

 

 

 

 

After that, we’ll create a Traffic policy which we’ll later attach to the LB vservers after which the web servers are located. Make sure SSO is turned on and you’ve selected the KCD account you’ve previously created (e.g. kcdaccount1).

NetScaler - Traffic Profile

There, all set for KCD on the NetScaler!

Configure NetScaler Unified Gateway

So, one of the cool new features is the Unified Gateway. Technically speaking the Unified Gateway allows us to put the NetScaler Gateway behind a Content Switch. The Content Switch is configured with a public-facing IP and accepts all client connections. The NS Gateway is then configured with a non-adressable ip (0.0.0.0). The CS redirects traffic to the NS Gateway based on a carefully constructed CS policy.NetScaler - Unified GatewayWe can even use the NS Gateway auth server for authentication (even to web applications), so no AAA vServer is necessary, which means we save a public IP!

And if you configure the Content Switch with a wildcard certificate (or a SAN certificate), you can put any host behind the CS (e.g. sharepoint.contoso.com, owa.contoso.com, …) using additional CS policies.

The easiest way to setup a Unified Gateway on NetScaler is to use the built-in wizard.

When creating the Unified Gateway, do note that the Unified Gateway IP you have to enter actually is the IP of the public facing Content Switch.

As for the certificate, best to use a wildcard certificate for your public domain, or a SAN certificate containing all external URLs (though less flexible if you want to add more URLs behind it). If you’re only going to use cVPN, then a standard SSL certificate may suffice – though since I only recommend cVPN for but the simplest use cases – I strongly advise not to go with that path.

Curiously you cannot select SAML for authentication via the Wizard, for now, add something like LDAP auth. That doesn’t have to work – we’ll disable authentication later on the NG vserver.

Portal theme and applications aren’t important (unless you’re going for the cVPN route).

Once you’ve created the Unified Gateway you’ll hopefully find a new CS and a new NG vserver on your NetScaler.

Let’s configure them for our solution:

First, make sure you add the SAML auth policy you’ve previously created to the NetScaler Gateway virtual server. You do this by editing this NS Gateway. Remove any other authentication policies which might have been added with the Wizard.

NetScaler - Gateway Authentication

NetScaler - Gateway settingsSecond,

a) switch authentication off (how weird that might sound)

b) enable Login Once and

c) enable ICA Only (if you’re not going to use SSL VPN or SmartAccess policies) so you don’t consume Universal Licenses.

 

 

 

 

 

 

 

 

 

Add your StoreFront configuration as usual (STA servers & Session Policies).

That should be it for the NS Gateway.

Configure StoreFront for Kerberos Contrained Delegation

StoreFront will be a special case. In order to have SSO to XenApp, we’ll not be doing KCD from the NetScaler to StoreFront. Actually, we’re going to do Protocol Transition to Kerberos on the StoreFront server itself.

For that we’re going to (supposedly) use Smart Card authentication on the gateway. In StoreFront, add (or edit) a gateway and make sure the Logon Type is set to Smart Card. Like always, make sure that your StoreFront server can perform the callback (check that it can resolve the URL). In our environment we’re using the following settings:

StoreFront - NetScaler Gateway

 

Make sure you also enable Smart Card authentication on StoreFront.

StoreFront - Authentication Methods

Also, on the store we are going to enable Kerberos Delegation:

StoreFront - Store Configure Kerberos Delegation

Configure the XML Broker to accept Kerberos tickets

Now of course, this also requires our XML brokers to speak Kerberos. In XenApp 6.5 this is only  possible if you share the XML service with IIS. IIS will then handle the authentication part (and IIS understands Kerberos).

This means that we need at least one XenApp XML broker where the XML service shares it’s port with IIS. If you haven’t chosen this option during the installation of your server, you could follow CTX125107 – How to Configure XML Service to Share Port with IIS on Windows Server 2008 to reconfigure the XML broker to share the service with IIS.

However I strongly recommend against performing the above because I’ve encountered several issues with this (like no applications returned from time to time, config gets broken when installing Rollup Packs, etc), so I advise to install new XenApp servers with the XML broker integrated with IIS, and to use them as dedicated XML brokers.

Make sure that the “CtxAdminPool” and the “CtxScriptsPool” application pools run under the LocalSystem account as per CTX130480, otherwise you also might encounter connection problems. This is necessary because we’ll delegate rights to the StoreFront computer account to perform KCD to the XML broker. By running the app pools under the LocalSystem account, we can add delegation to the HTTP service on the computer account of the XML broker.

Talking about Kerberos delegation, we must configure delegation as follows to allow SSO to XenApp:

On the StoreFront computer account(s) add delegation for:

  • HTTP service of all XML brokers (also check if an HTTP SPN record exists for the service on those computers with SETSPN -L <hostname>)

On the XML Broker computer accounts add delegation for:

  • CIFS & LDAP services of all Domain Controllers
  • HOST service of all XenApp servers in the farm
  • HOST service of itself
  • HTTP service of itself

Active Directory - Delegation XML Broker - 2

On the XenApp computer accounts add delegation for:

  • CIFS & LDAP services of all Domain Controllers
  • HOST service of itself
  • HTTP service of all XML brokers

Note that when using KCD, you must delegate the whole chain. That means for example that if you are using roaming profiles on a remote file server, you must also add delegation to the CIFS service of that file server on the XenApp computer accounts.

Next to all this, your XML Broker must also be configured to trust XML requests. The easiest way to do this is with the XenApp policy “Trust XML Request Sent to this broker”, which you then apply to the XML brokers.

Configure web resources to accept Kerberos tickets

When accessing web resources we will be using KCD on the NetScaler. Very important to note that NetScaler will use the server name (which you define via Load Balancing > Servers) to perform Kerberos SSO. This means that you need to add the server with the exact name of the service (which has an SPN record) you’re accessing.

For example, you’re running a website which is running in an app pool with the LocalSystem account (like the XML brokers), then:

  • you need to add the server with the exact hostname of the server hosting the website
  • on the service account you’re using for KCD, you need to add delegation for either HOST/<hostname> or HTTP/<hostname> on that server

In another example, you’re running multiple websites on one host (each with their own hostname) and one of those websites you want to perform Kerberos SSO for is running under a specific user account. Then:

  • you must ensure that a SPN record exists for that hostname on the user account on which it is running (e.g. if the hostname is www.contoso.com, and it’s running under the CONTOSO\svc_website account, then check if the SPN is registered by running “setspn.exe -L CONTOSO\svc_website” – this will list all SPNs for that account – if not, add the SPN with “setspn.exe -a HTTP/www.contoso.com CONTOSO\svc_website”)
  • add the server on the NetScaler with the hostname of the website. This could pose a problem when you’re adding servers by IP on the NetScaler, because NS only allows one server with that same IP address. A workaround is to add the server by DNS name. (if the NS can’t resolve it then, add an A record on the NS DNS).
  • on the service account you’re using for KCD, you need to add delegation for HTTP/<hostname> on <user account> (e.g. HTTP/www.contoso.com on CONTOSO\svc_website)

Active Directory - Service Account - 2

Then, add the traffic policy you’ve previously created on the LB vserver, that will ensure Kerberos SSO will be performed.

Inspiration / References

Closing notes

I’m aware I didn’t include the configuration for the claims based auth to SharePoint scenario (and how we still get SSO), but this is because it’s rather straightfoward. SharePoint, while load balanced with NetScaler, is just configured for Claims based auth, and uses the ADFS server as IDP. The LB vserver on the NetScaler does not perform any authentication. When a user wants to access SharePoint for the first time, he/she authenticates at the ADFS, after which AFDS sets its own session cookie. So when the same user later wants to access XenApp, and gets redirected to ADFS by the NS, ADFS reads the session cookie and performs SSO.

That concludes it for now. While not really easy, it’s definately very awesome, showing the power of Citrix NetScaler once again. Any questions? Use the comments below 🙂

Scheduling Process Monitor to run at a certain time

When troubleshooting an issue for a client of mine, I was in need for some further debugging information. To look into what processes are doing in the background, I turn to Sysinternals’ Process Monitor – one of the tools I recommend to master as an IT consultant. What was troublesome however, is that the issue I was looking into only occurred at night and I didn’t feel much for staying up late to fire off Process Monitor. To complicate matters, the systems themselves were in the production environment, and since Process Monitor does have a (hefty) impact on system performance (and the logs it creates grows very large very quick) I couldn’t just start it before leaving for home, leaving it running until I came back to work the next morning.

So it would be cool if it was possible to run Process Monitor as a scheduled task, run it for a few minutes (when the issue occurs) and retrieve the log the next morning to analyze it.

So, what options do we have when running Process Monitor?

Arguments

Looks like we have enough arguments to choose from to get this to work. So, let’s put the scheduled task together.

Create the scheduled task to run Process Monitor

  1. First, download Process Monitor if you haven’t already done that, and save it on the machine you want to monitor.
  2. On the machine, go to the Administrative Tools and open the Task Scheduler console.
  3. In the left pane, click on Task Scheduler Library
  4. On the right pane, click on Create Basic Task…
  5. Enter the name of the task (e.g. Start Process Monitor) and click on Next1. Create a Basic Task
  6. Choose the frequency. Since I only want to run it once, I chose “One Time”. Click on Next2. Trigger
  7. Enter the time when you want the task to be run (before the issue occurs of course) and click on Next3. One Time
  8. For the action choose “Start a program” and click on Next4. Action
  9. Browse for the Process Monitor executable (procmon.exe), and add /accepteula /quiet and /BackingFile <log path> (e.g. C:\temp\log.pml) as arguments. Make sure you have enough diskspace where you’re saving the log file.5. Start a program
  10. Check “Open the Properties dialog…” and click on Finish6. Finish
  11. The properties dialog is shown. On the bottom, click on “Change User or Group”, select SYSTEM and click OK. Click on OK again to close the properties window.7. Security options

Great! We’ve set up the scheduled task! However, if we do nothing else Process Monitor will run on forever, and we don’t want that to happen… Unfortunately, there’s no argument to run PM for a certain amount of time. You could, using the options of the scheduled task, kill the task after a certain amount of time. However, when you just kill the process and don’t shut it down properly, the log will get corrupted, so we can’t use that. Fortunately, we can run Process Monitor with the /terminate option, which will terminate all running instances of PM (running in the executing user’s context). So, we’ll set up a second scheduled task to terminate PM after a set amount of time.

Create the second scheduled task to stop Process Monitor

  1. Open the Task Scheduler console again.
  2. Create a new Basic Task
  3. Enter an appropriate name (e.g. End Process Monitor)1. Create a Basic Task
  4. Choose the frequency.2. Trigger
  5. Enter time time to end Process Monitor. The should be after the issue occurs (and inherently later the time when starting PM).3. One Time
  6. For the action, choose “Start a program” again.4. Action
  7. Select the executable again, but enter /terminate as the argument.5. Start a Program
  8. Check “Open the Properties dialog…” again and click on Finish6. Finish
  9. In the properties window, change the user to SYSTEM again in the security options.7. Security options

And there you have it! The next morning (after grabbing a cup of coffee) you’ll be able to open up the log file and analyze the issue.

Hope this was helpful!

0000007B STOP error after V2P

Because of certain issues with XenClient on my laptop (external webcams half working, battery losing power before the system completes hibernation, wireless with computer based certificates not working, sluggish hard disk performance…), I decided to do a reverse P2V, a V2P (virtual to physical) of my main virtual machine (Windows 7 Enterprise).

First, I uninstalled XenClient tools to make sure the XenClient drivers were removed – so they couldn’t cause any trouble. Then I bought a copy of Acronis TrueImage and imaged the virtual machine and stored the .tib file on my NAS.  It took about 5 hours to do so (did I mention sluggish hard drive performance yet?), but it did the job.

Then, using a TrueImage boot cd, I restored the backup to the disk. I selected the three partitions, well actually two – the system disk (containing the Windows folder) and the System Reserved partition (I’ll come to that in a sec), plus the MBR (Master Boot Record).

When the restore was complete, I could boot Windows (yey), however during boot it BSOD’ed with a 0000007B STOP error. Booting into safe mode didn’t work either. Bummer. At that moment I was thinking that the System Reserved partition was causing this issue. The System Reserved partition is by default a small (100MB) partition containing System Recovery tools and is also required for BitLocker – but isn’t really necessary for Windows to work. When present, this is also the Active partition, and hence contains the boot files. So, if this somehow gets corrupted, it can cause booting to Windows to fail.

After some hours of troubleshooting with bootrec and bcdedit, I decided to remove this partition altogether. I had to set the system partition as the Active partition and run bootrec /RebuildBCD in order to allow booting from the system partition without the System Reserverd partition.

Still, booting into Windows resulted in a 0000007B error. Now with only a single partition left, I was growing convinced this was a driver issue. Since the OS was virtualized it was working with a different set of drivers and didn’t have the drivers for the bare metal hardware on board. Since the laptop was based on a Intel chipset, I thought a about somehow installing the Intel RST drivers. However, this deemed impossible, so that was a dead end.

Then after some searching on Google, I hit a MS KB (922976 – Error message occurs after you change the SATA mode of the boot drive), which I thought could be related somehow. Now, I knew that in my laptop BIOS the disk access method was set to AHCI. Maybe the VM, being virtualized, was accessing it using ATA? So, the article describes setting the Start value in the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Msahci key to 0 as a solution. Since I couldn’t boot into Windows, I used a Windows 7 DVD to go into System Restoration mode and launched a Command Prompt. From there I launched regedit, loaded the SYSTEM hive on disk (C:\Windows\System32\Config\SYSTEM), navigated to ControlSet001\Services\Msahci and noticed Start was set to 3. I changed it to 0, unloaded the hive and rebooted.

And Windows 7 booted to the logon screen! Succes! From there it’s just a case to install the remaining drivers to have a fully functioning Windows installation. Hopes this saves others a lot of time and headaches!

In any case I wish all a happy New Year!

Troubleshooting “An error occurred while making the requested connection”

A dreadful error for users, but for admins too. Though, usually it’s easily fixed (in most cases there’s a problem with the load or licensing), but not in this particular case I encountered.

Well, one of the first things to do is to check out the event viewer of the Web Interface. I noticed I encountered a whole lot of 30102 errors coming from Citix Web Interface:

Error 30102

What’s this error about?

30102 Error Details

Now there can be a lot of things that can be causing this error. What’s of importance are the XML server which is queried and what function is reported to cause the error. The latter is [com.citrix.xml.NFuseProtocol.RequestLaunchRef].

The first thing I tried is restarting the XML service on the reported server (usually this is the top server you entered as server in the Citrix Farm (Web Interface > Server Farms), unless you use the list to load balance the XML request), but that didn’t help.

I tried various other things, but eventually I fired up WireShark, my tried and trusted networking tool, on the Web Interface server. I captured all traffic, but filtered out traffic coming from and going to the XML server (filter: ip.src == <ipaddress> || ip.dst == <ipaddress>), and looked at packets which were flowing when the 30102 errors occurred in the event viewer. I found the following issue:

POST /scripts/wpnbr.dll HTTP/1.1
Content-Type: text/xml
Host: xxxxxxx:9090
Content-Length: 561
Connection: Keep-Alive <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE NFuseProtocol SYSTEM "NFuse.dtd"> <NFuseProtocol version="5.4">     <RequestLaunchRef>         <LaunchRefType>ICA</LaunchRefType>         <TicketTag>IMAHostId:15274</TicketTag>         <DeviceId>WI_QZBNv5ztEDpLu0THQ</DeviceId>         <SessionSharingKey>-peML5lFv4meHDULuC8Y14e</SessionSharingKey>         <Credentials>             <UserName>xxxxxxxx</UserName>             <Domain type="NT">xxxxxxxx</Domain>         </Credentials>         <TimetoLive>200</TimetoLive>     </RequestLaunchRef> </NFuseProtocol>
HTTP/1.1 100 Continue
Server: Citrix Web PN Server
Date: Fri, 25 May 2012 09:22:12 GMT
HTTP/1.1 200 OK
Server: Citrix Web PN Server
Date: Fri, 25 May 2012 09:22:12 GMT
Content-type: text/xml
Content-length: 217 <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE NFuseProtocol SYSTEM "NFuse.dtd"> <NFuseProtocol version="5.2">     <ResponseLaunchRef>       <ErrorId>unspecified</ErrorId>     </ResponseLaunchRef> </NFuseProtocol>

Eventually, I found out that every time this happened the same TicketTag (in this case IMAHostId:15274) was given. When the TicketTag was different the response gave a valid ticket:

HTTP/1.1 200 OK
Server: Citrix Web PN Server
Date: Fri, 25 May 2012 08:06:29 GMT
Content-type: text/xml
Content-length: 228
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE NFuseProtocol SYSTEM "NFuse.dtd"> <NFuseProtocol version="5.2">     <ResponseTicket>       <Ticket>11CE036B24379D427366DA81E873C0</Ticket>     </ResponseTicket> </NFuseProtocol>

So it must have something to do with this particular TicketTag. Now, this IMAHostId responds to a specific server. When you go look on a XenApp server at HKLM\SOFTWARE\Wow6432Node\Citrix\IMA\RUNTIME\HostId you find the specific IMAHostId of the server (in decimal).

So putting all this together, it seemd that getting the LaunchReference from a specific server was failing. When I found the server by the IMAHostId (I just made a quick & dirty script to echo the HostId’s of all the servers in the farm), I restarted its IMA service, et voila, LaunchRequests to the server worked again!

(Eventually I found out it was not one, but two servers which were showing this behaviour, so I had to restart both IMA services. Anyway, I’m glad that this issue was solved!)

Happy troubleshooting! 🙂

UPDATE: On a recent project I’ve encountered this issue again, though this time it wasn’t the IMA service which was in trouble, but the XML broker didn’t trust XML requests sent to it. Enabling “Trust XML requests” Citrix computer policy on the XML brokers was the fix for this.

Changing the server certificate on Citrix AppController

Not so long ago I set up the new Citrix AppController (part of the new Citrix Cloud Gateway Enterprise) in a test environment to get to know the product a little better. Setting up the server wasn’t too much of an hassle, albeit not much documentation is available. I successfully added some web applications, and SSO worked.

What did annoy me however that I still got the red address bar in my browser, indicating that the website is insecure. By default, the AppController uses a self-signed certificate – which is, ofcourse, untrusted. So there are two ways to solve this; the simplest one is just to trust the self-signed cert by importing it. The other is to create an official certificate to be used. The problem is that the documentation on edocs.citrix.com is flawed, and will not work. So I had to figure it out myself. That’s what this post is about.

I’m not going to elaborate on how certificates work, I assume you have some working knowledge on the matter. To create the certificate we follow three steps:

Create a certificate request for the FQDN you’ll be using for the AppController

  1. Log on to the console of the AppController (eg with Putty)
  2. Remove the current private key
    sudo keytool -delete
    -alias tomcat -storepass changeit -keystore /root/.keystore
  3. Create a new key
    sudo /usr/java/jdk1.6.0_21/bin/keytool -keysize 2048 -genkey -alias tomcat -keyalg RSA -storepass changeit -keystore /root/.keystore
    You’ll be asked to fill in the details for the cert request. Important: When asked for “What is your first and last name?” enter the FQDN of the AppController, not your real first and last name!

Request a certificate from your Certificate Authority

  1. Create a certificate request
    sudo /usr/java/jdk1.6.0_21/bin/keytool -certreq -keyalg RSA -alias tomcat -file /home/admin/certreq.csr -storepass changeit -keystore /root/.keystore
  2. Use a SCP client, like WinSCP, to download the certificate request (/home/admin/certreq.csr) from the AppController onto your PC.
  3. Use the certificate request (it’s formatted in Base64 BTW) to request a certificate from your CA (your enterprise CA or a public CA). Download the certificate in PEM/Base64 format.

Installing the certificate

  1. Upload the certificate chain (make sure they’re all in PEM/Base64) to the AppController. Do this again with your SCP client. Upload the certificates to /home/admin/
  2. Import the root and intermediate certificates with
    sudo /usr/java/jdk1.6.0_21/bin/keytool -import -trustcacerts -alias root -file /home/admin/<root certificate> -storepass changeit -keystore /root/.keystore
  3. Finally, import the server certificate:
    sudo /usr/java/jdk1.6.0_21/bin/keytool -import -trustcacerts -alias tomcat -file /home/admin/<certificate> -storepass changeit -keystore /root/.keystore
  4. Reboot

There, now you have that nice white or green address bar 🙂

Editing Farm Policies using the XenDesktop 5 SDK

Editing the XenDesktop Farm policies with PowerShell is not that hard, if you know how. I couldn’t find much, if any, documentation about the subject (only some for XenApp 6), so here’s a quick how to:

First, you start with adding the necessary Citrix PowerShell modules with (you perform this on a XD Controller on another machine with the XenDesktop SDK installed):

Add-PSSnapin Citrix.*

For convenience, we’re going to map the XDFarm policies to a drive, so we can browse it easily:

New-PSDrive XDFarm -root \ -PSProvider CitrixGroupPolicy -Controller <FQDN of a XD Controller>

From there you can browse the XD Farm policies easily:

Set-Location XDFarm:\
PS XDFarm:\> Get-ChildItem
User
Computer

If, for example, you would like to add multiple session printers to the Unfiltered User policy you would perform the following:

Set-ItemProperty XDFarm:\User\Unfiltered\Settings\ICA\printing\SessionPrinters
-name Values
-value @("<pathA>,model=<modelA>,location=<locationA>","<pathB>,model=<modelB>,location=<locationB>")

So, there you have it 🙂

Do note that you’re working with an in-memory version of the CitrixGroupPolicy object. That means that if you change settings in the Citrix Desktop Studio, this won’t be reflected in the current instance. Changes that you make however will be automatically written back to the XenDesktop database.

Controlling information

I’m putting some more time in to properly set up my blog. Things yet to do is create an own theme… and well, create more blogposts 🙂

One important thing is how to control information in the world of Twitter’s, Facebook’s and Google +’s of this world. As you know, these services rise and fall, and when they fall, your content does too. Then there’s obviously features that are redundantly provided by many of such services, like giving a status update. What’s also annoying is that you need to update each and every service (why are you using them otherwise?) and that becomes a chore quickly.

So what I’m about to do, is to centralize everything in my personal WordPress blog. It’s hosted by myself, so it’s my content and I can do anything I’d like to do with it. And to make life easier, I’m going to use the Twitter Tools plug-in to automatically tweet updates. In Facebook, I’ve integrated the Twitter app, so tweets should also show up as Facebook update. So yeah, it’s kind of experimental 😉

Let’s see how this works out…