In my last post, I explained how all of the HPC macros are executed on the desktop, except for the HPC_Execute macro. I cannot emphasize enough how important this is to design. The HPC_Merge macro must be optimized to be as fast as possible. Otherwise it will become a bottleneck. The reason is that this macro is called after every core completes its iteration. If you have 100 compute nodes, then the HPC_Merge macro will be called 100 times between “cycles”. The more compute nodes that you have, the faster the HPC_Merge macro needs to be.

Let’s look at some examples.

In a hypothetical good situation, we have 100 compute nodes that each take approximately 20 seconds each to run. When complete, each sends data back to the HPC_Merge macro which take 0.05 seconds to run. This works out to just 5 seconds of merge time per “cycle”. We should not encounter much of a log jam. Each complete “cycle” will take roughly 20 seconds. So if we are computing 10,000 iterations, this means 100 cycles of 100 nodes. 100 x 20 = 2,000 seconds or 33 minutes.

In a hypothetical bad situation, we have the same setup as above, but the HPC_Merge macro now takes 5 seconds to run. This equals 500 seconds per “cycle”. The HPC_Merge macro is now the bottleneck. The performance of the compute nodes is somewhat irrelevant. Each cycle will now take approximately 500 seconds. So, again, if we are computing 10,000 iterations, this means 100 cycles of 100 nodes. 100 x 500 = 50,000 seconds or 833 minutes or roughly 14 hours.

Notice how an inefficient HPC_Merge macro can absolutely destroy the benefits of cluster computing.

You MUST optimize the performance of this macro. Performance issues in the other macros will not have a huge impact, or can be easily overcome by scaling out the compute nodes. You have limited options on the HPC_Merge macro other than to optimize the code.

There are three primary ways to speed up the HPC_Merge macro:

1. General VBA optimization techniques such as turning screen updating off
2. Using in memory data structures rather than storing things in cells
3. Optimizing database connections (if you have any)

I will cover each of these in future posts.

This is a game changer.

In order to take advantage of HPC in Excel 2010, you must do a little up front work first.

Step 1: Get a cluster set up so that you have an environment on which to run. There are a number of ways to accomplish this task. For an overview, see my previous post: How to Get Started with Excel on the Cloud.

Step 2: Set up a shell workbook with the proper references, macros, and system variables defined. Microsoft’s website has an excellent White paper outlining exactly what you need to do in order to get this set up. I highly recommend reading this paper in detail and then use one of their sample workbooks as a template.

Step 3: Restructure your macros to interact properly with the standard HPC macros. The rest of this post will focus on this step.

Structure

The standard HPC functions are:

1. HPC_Initialize – Performs any up front calculations or set up activities
2. HPC_Partition – Determines how to split the calculation into parts
3. HPC_Execute – Perform a single iteration on the compute nodes
4. HPC_Merge – Gather results from the compute node
5. HPC_Finalize – Process wrap up activities once all iteration of the HPC_Execute macro have been merged.
6. HPC_ExecutionError – Handles calculation errors
7.  HPC_GetVersion – Returns the version of the HPC macro framework being used.

In order to maximize the performance gains using HPC you must really understand these macros and their implications.

Probably the most important thing to understand is that all of the macros except for the HPC_Execute macro run on the desktop. What this means is that you need to push as much of the processing as possible to that macro in order to get any noticeable benefit.

Data Flow

It is also very important to understand the data flowing from the desktop to the compute nodes and back again. Transferring data can be very slow and is often the bottleneck in applications.  There are some basic rules to follow:

1. You will want your desktop to be located as close to the compute nodes as possible. If using Amazon EC2, this means using a remote desktop.
2. You need to minimize the data flowing to and from the compute nodes. If possible have the compute nodes write results to a fast database. This is much faster than sending the results back to the desktop. The bottleneck is the HPC_Merge macro. This runs on the desktop whenever a compute node finishes an iteration.
3. Use fast data structures and pass data in bulk rather than passing values one at a time.

I will cover data flow in a lot more detail in future posts.

Partitioning Strategies

Because of the speed issues related to transferring data from machine to machine, it is necessary to sometimes think about the right way to partition up your jobs.

Using our earlier example of 20 eight-core CPUs, we know that we have 160 cores at our disposal. Let’s assume we want to run 160,000 scenarios of a model where the calculations are performed for each of our 16,000 customers. Therefore we have 2,560,000,000 loops to process. There are a number of ways we could accomplish this goal.

1. Send all 16,000 customers along with individual scenario data to 160 cores at a time 1,000 times.
2. Send 100 customers along with individual scenario data to 160 cores 160,000 times.
3. Send 16,000 customers and scenario data for 1,000 scenarios to each of the 160 cores 1 time.
4. Send 16,000 customers and scenario data for 10 scenarios to each of the 160 cores 100 times
5. etc.

You have to realize that each time the compute node runs, the entire Excel workbook and all of the needed data gets transferred across the pipe. As such you need to optimize in four dimensions:

1. Workbook size – smaller is better
2. Input data size – smaller is better
3. Output data size – smaller is better
4. Memory usage – smaller is better

This is easier said than done, unfortunately. Workbook size and input data size are very related to each other. Since the workbooks all get copied over each time, you need to try to shrink as much as you can. Workbook size and memory usage are very related to each other. As workbooks grow (sometimes by storing temporary results) they take up more and more memory. If they take up too much, then the processing speed will degrade. Output data size and memory usage are related the same way. Often we can shrink the memory usage by sending back results instead of keeping them in memory or in the workbook.

As you can see it is a bit of a balancing act and will often take a bit of experimentation to structure the partitions the right way. These decisions will be influenced by how much data is flowing around, how powerful your cores are, how fast your network is, etc.

I will go into all of this in more detail in future posts.

Wrapping up

You should now be getting more familiar with what is possible with Excel 2010 and HPC. Please follow along over the next few weeks as I dive into all of this in a bit more detail.

If you have any questions or need any help, please leave a comment.

1. Use an existing cluster on your network
2. Build a small network/cluster yourself
3. Build a cluster using Amazon EC2
4. Use a third party service

Use an Existing Cluster

If you already have a cluster available then this is by far the easiest option. You just need to get the details from your network administrator.

Build a Network

Another option, though a bit over the top in my opinion, is to purchase a few machines and build the network and cluster yourself. Microsoft put together a nice tutorial video if you are so inclined. If you are in a corporate environment, this could be very tricky…

Build a Cluster Using Amazon EC2

If you want many servers on the cheap, then it is hard to beat Amazon EC2. The EC2 service is very inexpensive, is very scalable to almost any size, and is fairly easy to use (once configured). The bad news is it is not so easy to configure. If you are not a network administrator, then you may find the process a bit overwhelming and time consuming. When I did this the first time it took me a few days…

Detailed instructions for setting up the cluster are available from Amazon. Be warned, these instructions do not consider that you need Windows HPC Server 2008 Enterprise in order for Excel 2010 to work properly with HPC. You can download a free 180 day trial from Microsoft. Detailed instructions on setting up for Excel 2010 HPC are available from Microsoft.

If everything is set up properly you should be able to run the sample files provided with the Excel 2010 instructions above.

Please note that you will need an Excel 2010 license for each compute node.

Use a Third Party Service

Given the complexity of the above methods it is no wonder that service providers have popped up to assist. Providers include Nimbis and MicroHPC.   Using a provider can get you up and running in very little time. However, be aware that all pre-canned solutions always have their own limitations.

Next Steps

Once you have the cluster up and running and have been able to run the sample workbooks then it is time to kick it up a notch and get your hands dirty. In my next post, I’ll walk you through the details of how to convert a standard workbook into an HPC ready workbook to allow you to take full advantage of the power of HPC.

As a result, we have moved away from very transparent models to computer models that are anything but. The unintended consequence is that these models have become increasingly difficult to understand or to check.

Whole “sub-models” are hidden away and buried from view. Those now using these models often do not even understand what these models are doing behind the scenes. Computers now are often just blindly trusted.

This need for trust has been built up as the logic of the models has been hidden further and further away from the user of the models. This was somewhat necessary back when computers were much slower. Logic was programmed by professional programmers and run on mainframe computers. This situation was far from transparent. In order to see the details of calculations and understand the models, analysts needed to rely on printed output generated by these behemoth systems. This situation caused a lot of uneasiness for quite some time and continues to do so today.

Enter Lotus 123 in the mid-eighties. Analysts could now model fairly complex situations without the need for programmers. The struggle between the business and IT truly began. PC based models were fairly limited in scope due to the limited processing power of the PCs themselves. Over time Microsoft Excel overtook Lotus 123 by its ease of use and by adding much more powerful functions than were available in Lotus.

The real turning point was the release of the Intel Pentium chip combined with the release of Excel 5 (aka the birth of VBA) in 1993. All of the sudden PCs were relatively fast and business users could now really get their hands dirty with very easy to write code.

Business users have since fallen in love with Microsoft Excel. Anyone with even the slightest bit of programming ability can develop very complex models in a platform that is incredibly transparent and easy to check (if constructed well). This has caused a number of real problems for the IT department that is often asked to inherit “tools” written in Excel after the tools are being used by many users. These tools are often not fit for enterprise use.

I think we are at another tipping point similar to 1993. Microsoft has once again created a game changer with Excel 2010. This latest version of Excel now allows end users to easily write macros that can be executed in parallel using clustered computing. At the same time various vendors are rolling out cloud computing resources for the masses. If you pair Excel 2010 with Amazon’s EC2 service then you have an incredibly powerful incredibly cheap super computer. The modelling possibilities become literally endless.

Excel models that used to take days now take hours. Those that took hours now take minutes. The trick is finding the right price point. For a price you can make these models as fast as you would like.

What I cannot figure out is why no one seems to know about these features. There is very little information available and very few companies have started to assist in setting up the cloud resources needed.

For large corporations, this is a good thing. It is now time to re-think how Excel fits in the enterprise. It is no longer possible to control the business by limiting access to IT. The proverbial cat is out of the bag. Now that the IT department is not needed in order to build and execute very large, very complex models, the business must be more controlled.

This is going to be a pain for everyone involved. Given the current regulatory environment, auditors are taking very hard line stances regarding control. Most Excel users are anything but controlled. With power comes responsibility. Compliance will not be able to stop users from building these tools. End user computing policies must be crafted and strengthened to meet the needs of both auditors and end users. End users must be trained on how to build controlled, auditable tools.

Welcome to the new world order.

1. the real technical expert conveys their requirements to an operations analyst
2. the analyst then conveys the information to a business analyst in responsible for IT
3. the IT analyst then writes it down for developers and QA staff
4. the developer then tries to understand what is needed
5. QA then tries to sign off

The people doing the work and signing off are far removed from the business requirement owner. By the time the requirements get to the developer the requirements have been interpreted far too many times. Questions typically go to resources that are not the true experts in the field. In the end the requirements communicated sound more like “I think the business needs something kind of like this, I think”.

The QA staff are equally hindered in their job. Rarely do they actually know what the software being built is actually supposed to do. They are often unaware of the big picture and typically do not have the business skills to know if the big picture need is being met. Their role is typically abstracted to the point of simply ensuring that the code matches the specification.

This is a key point especially when using standard QA testing platforms as the requirements are broken down into many individual parts and each part passes or fails testing. QA tools are not well suited for determining if the overall requirement is being met.

All of this process is painful, time consuming and error prone. This is magnified greatly when using off-shore teams. As a result, the process developing new products often fails to deliver as promised. IT is repeatedly blamed for falling ever behind on deadlines, blowing budgets, and worst of all delivering tools that are insufficient to do the job.

It is critically important to get the business closer to the code and to remove as many layers as possible between the business owner and the end product.

This can sometimes be achieved by hiring business experts within the IT department. In large IT shops this can pay big dividends. These experts play a key role by being able to quickly clarify requirements that are unclear or open to interpretation.

In the off-shore situation, it is critical to get business expertise in the off-shore shop. The language barrier combined with often limited business knowledge means that off-shore teams tend to miss subtleties in requirements and generally have little concept of what tools are really trying to accomplish. The time zone difference exacerbates the problem often causing severe delays. In large shops the cost of embedding experts more than pays for itself.

On the far end of the spectrum is the solution of removing the need for IT resources all together. The ideal situation is when IT delivers an infrastructure to the business that allows the business to:

1. Code their own business logic using tools that they are familiar with (like Excel)
2. Have complete visibility to the programming logic allowing them the easily sign off
3. Have the ability to deploy routine changes without the need for heavy deployment activity.

This is all very possible using modern tools widely available in the marketplace. The key complication is building in the appropriate controls to ensure that the logic changes have been properly tested and signed off.

None of this is terribly difficult technically. The difficulty lies in two areas:

1. Architectural design needs to accommodate such activities
2. Governance and control policies need to be adapted to allow the business to be more involved.

IT then becomes the enabler that businesses are demanding and the business is able to more precisely and actively manage critical business processes and logic. When executed well the results can be dramatic.

The current regulatory environment in all corners of the world is becoming more complex by the day. This is particularly true in the US. The scale of mandatory regulatory change can often overwhelm the available IT resource supply. The mandatory nature of the regulatory change often causes even the most important business priorities to get delayed, often indefinitely.

The business demand for IT solutions is growing at an exponential rate. Unfortunately business leaders are ill-equipped to quantify the value of IT. Since many solutions are internal and are not sellable products, the return is often very opaque. As a result, Finance tends to squeeze IT budgets to the point that IT cannot deliver the services that the business leaders are adamant they need.

Even when budgets and resources do exist for key business objectives, the IT processes and the onus of compliance typically greatly lengthen project timelines. The processes are typically so rigid that even the mention of the word “change” can instil fear in the project manager. The business environment can evolve dramatically during the timescale of a typical large project. Current standard IT processes cannot cope with this reality. As a result most IT projects are deemed failures. The problem is not with IT, but instead with the rigid controls that are in place preventing them from meeting the needs of the business.

 I would like to argue that the solution is rather simple. We need to remove the need for IT resources as much as possible.

 Accomplishing this goal is not easy, primarily because it requires careful consideration, careful design, and an acute understanding of the business and its environment.

In the ideal solution, each key system has the following characteristics:

1. Compliance and governance would scalable depending on the nature of change
2. Business logic would be owned and maintained by business owners but subject to governance
3. Business logic would be defined by the business using tools that they understand and are comfortable with (ie. MS Excel)
4. Business logic changes would be deployable by business owners in a similar manner as parameter changes are typically made, without the need for large scale IT deployment/change control processes.

You should notice that the key theme of the above solution is to make IT an enabler for the business. The tools are designed to allow the business to manage itself. The current situation is much different.

In response to the push towards compliance, organizations have lost all common sense.Compliance tries to prove beyond a shadow of a doubt that all change has been tested and documented to death. Unfortunately, the manner in which this is implemented insures that the change is tested and documented primarily by someone who knows very little about what it is that they are signing off.

A change is needed to put the control in the right place, in the business. The business owners are often viewed with contempt by most folks in IT because of their “cowboy ways”. I would argue that the behaviour of the business is in response to unreasonable control environments. To break this vicious cycle, compliance, IT and the business must find a better way.

Let the business own and maintain the business. Make them accountable for the compliance aspects. Empower them to do their job. Punish them if they don’t.

Once this change is made, the role of IT is not to make simple business rule changes. Instead IT is needed to change things that no one envisioned would change.

The mind-set of IT changes from “preventing the business from blowing things up” to “enabling innovation”. Which role would you rather play?

There are a couple of different angles to consider.

1. The front office needs to run and update models continuously sometimes multiple times a day. The standard IT process just does not and cannot work in that environment.
2. The back office is being confronted with new mandatory regulatory change at an ever increasing rate. The IT development cycle is typically so long that companies have a difficult time simply implementing required changes in the timelines necessary.

The Front Office Risk

The inability for IT to accommodate the front office poses real risk to the enterprise. Analysts often have no choice but to build complex tools using whatever resources they can bring to bear on their own. More often than not, the solutions built consist of very complex Excel workbooks that have not been thoroughly tested. These solutions are often hidden from the view of auditors as they lack any real evidence of control.

The Back Office Risk

Standard IT processes are generally there to ensure compliance with the controls in place for the day to day operations of the company. If the numbers MUST be right, then the software needed to produce the answers MUST go through the process. These processes become sacrosanct as companies acquire standards certifications such as SAS70 or the various flavors of ISO. The processes cannot be bent in any way as to put these credentials at risk. As a result IT tends to grind to a halt, change is glacial and the company’s ability to adapt is severely diminished. This is now the situation at most regulated organizations.

Existing Efforts at a Solution

Many people have tried to solve this problem and none have succeeded. The overwhelming push to “agile” development approaches proves that organizations are desperate to solve the dilemma. The problem with all current “solutions” is that they fail to address the real underlying problems.

How Do We Solve The Problem?

The real underlying problems that need to be addressed are:

1. Systems are designed poorly and provide little flexibility to the user
2. Controls are inflexible and are not fit for purpose
3. The IT specification and testing processes create too much distance between the subject matter expert and the underlying code.

Any solution needs to address these issues heads on in order to be successful. Once this is accomplished the IT resources can be truly “agile”. To avoid confusion with the current “agile” methodologies we need a new term. Let’s coin the term “Nimble IT” to describe this solution.

Over the coming weeks I intend to dive into this area to highlight exactly how Nimble IT is achievable in any organization and why it is imperative that organizations adapt Nimble IT processes as quickly as possible.

Unlike “agile development” this does not require wholesale changes in methodologies resulting in wholesale departmental rebuild. Instead, Nimble IT requires a focus on design from both an architectural and a UI point of view. Any development methodology will benefit from Nimble IT, so pick whichever fits your organization best.