Thomas Gathings

I saw an interesting comment the other day on the topic. My thoughts are that Segment Architectures are parts of the whole Enterprise, either functionally (Sales, HR), geographically (Europe, Canada), or perhaps by some other taxonomy.

Capabilities, however, are not constrained by Organizational or Functional Architecture. For example, ‘Electronic Signatures’ is a Capability that may be deployed in all of Canada but only in European Sales. To further the confusion, business functions (Marketing, Customer Support) may exists in your Capability Architecture views as well as your Functional Architecture views, where Electronic Signatures and Green Data Center, for example, are solely capabilities.

We subscribe to a research service, and they use what I feel to be a useful classification system for the different levels of support that a system may have for a specific feature.

From the research service:

Rather than simply asking vendors whether their system has specific features, we asked vendors to be much more specific about how features are provided in their systems. In many places in this knowledgebase, you will see the terms Automatic, Semi-automatic, Semi-custom, Custom, NA, and third-party add-on listed next to a feature. Here are the definitions of what these terms mean:

Automatic: Built-in, out-of-the-box functionality that can be simply switched on and ready to be used.
Semi-automatic: The feature is mostly available in the system, but requires some programming and/or customization to activate.
Semi-custom: This feature is partially available in the system and can be adapted through some moderate custom program.
Custom: This feature is not available in the current feature but can somewhat easily be added through custom programming services at the time of implementation.
NA; not a current feature: This feature is not available in the current release of the software.
Third-party add-on: This feature can be added upon implementation using third-party software

Amongst architecture types, there’s long been an awareness that simple is good, and simpleton is yuck. The ability to articulate an idea is particularly important in architectural disciplines where the subject matter is often so complex (we are talking about enterprise business, applications, data, and technology architectures) that it must be simplified.

Recently, I’ve seen a different abuse of simplification….’oversimplification’. This is defined as “To simplify to the point of causing misrepresentation, misconception, or error.” Obviously, not a good thing for the architecture.

There are many ways to boil the fat out of an idea while retaining the essence of the flavor.

Establish an Architecture Vocabulary
Unfortunately, posting 147 items to your ‘Architecture Definitions’ SharePoint list is not going to cut it. Teaching, repetition, affirmation, and a little patience (from both ends!) go a long way. Also, stay cognizant of the native domain language of the audience and clarify or qualify when terms overlap (Database pros and business people first think of something completely different when they hear the term “Constraint”).

Use levels of abstraction
Everyone uses levels of abstraction but they don’t always realize it. There are many schools of thought and even domain-specific levels…logical-physical, object-experience-concept, conceptual-requirement-implementation, system-subsystem-component and so on. Often others are thinking at different levels. With awareness, you can avert a misunderstanding.

Use a Metamodel
Models are the primary means in which architects communicate architecture, of course, but models are only as good as their understanding. Metamodels provide rules and structure as to how models can be created. A good metamodel will meet almost all of your needed viewpoints, but expect it to grow and change over time. ARIS provides some 300 model types out of the box and allows you to refine and extend them. Also, there are some generic model types that, while not ‘integrated’ with the rest of the architecture repository, do allow total flexibility to create any otherwise unsupported viewpoint. Discourage others from creating ad hoc models, instead using the opportunity to further promote the metamodel.

Understand and be upfront about the Implications
Everything in IT has implications…time, money, scope, and quality being the A-list party guests. Beyond those, there can be dependencies, constraints, gaps, risks, and assumptions. Perhaps stemming from a natural tendency to be risk-aware, I find the ‘downsides’ to be exactly 50% of the decision-making formula. When someone is selling an idea, force them to cough up the downside!

A colleague asked how we might go about modeling the current state infrastructure in anticipation of several enterprise-wide projects.  The projects will require new target state definitions and ensuing gap analysis.

From experience, I have learned that it is not feasible to preemptively create a complete current state model of the application, data, or technology domain to a degree in which all of your future project needs will be met; there are simply too many possible views.  To do so would require identification and classification of stakeholders, their concerns, and a careful and deliberate selection of relevant viewpoints.  That alone is no small task, never mind the modeling effort itself.  That said, there are a a number of views that may be worth creating (ahead of time) in our particular scenario.

One of them is what I call the ‘Architecture set’.  It needs a new name, but that’s a different story.  This viewpoint is expressed at 2 levels (requirement and specification) and is similar in nature to the TOGAF Technical Reference Model.  Our EA tool confines this model to IT architectures, though conceptually, it is similar to a capability map or service structure model.  

The requirement level shows the set of logical elements that conceptually comprise an enterprise architecture by means of organizing together related Architecture Building Blocks, such as ‘Operating System’.  The specification level shows the actual realization of conceptual elements, Solution Building Blocks such as ‘Windows XP’.  The Architecture Building Blocks in the requirements level model are assigned (linked to) to an instance of the requirement level viewpoint, specific to that block. The specification level is a matrix, and further decomposes the building block into more detail.

Level 1: Requirements. The lavender elements are high-level ‘Architecture Building Blocks’.

The main concern these viewpoints address is ‘what does the enterprise need and how are we fulfilling those needs’.  A second benefit is that, with placement of the Standardization Status attribute, the specification level stands in as our Standards Information Base.  This extension helps architects determine the standard and available building blocks with which to compose target architectures.

Level 2: Specification. The blue elements are application systems serving as ‘Solution Building Blocks’.

Patting myself on the back, I obtained the full TOGAF 9 certification last Friday.  It has been a long time since I have certified in anything, and this feels pretty good. 

What does this mean to me? For years, I have held the position that certificates prove very little, except that the certified could pass an exam…that this did not necessarily translate into an ability to deliver.  I held an axiom that success was nearly a guarantee for those with innate talent (‘smarts’) working in an environment that motivates (‘drive’).   While I love those ideals, I have to add knowledge to the foundation. 

Knowledge comes in two forms: tacit and explicit. Tacit knowledge is the kind associated with ‘working knowledge’, or ‘know-how’.  I think of it as a result of experience.  Explicit knowledge is more readily articulated, transmitted, and perhaps easier to identify.  Is it fair to associate explicit knowledge with study and tacit knowledge with experience? 

The TOGAF certification process considers each of these knowledge dimensions. Part 1 TOGAF Foundation is standard multiple-choice format: given a question, select the correct answer. Part 2 TOGAF Certified is scenario-based complex multiple choice: given a scenario, select the best answer.  While the exam doesn’t prove an ability to deliver, it certainly tests one’s ability to apply knowledge in a way that demonstrates “know-how”.

Being intelligent and motivated are non-negotiable, but knowledge is key, too. Certification demonstrates some degree of knowledge. For me, the certificate was a great motivator, even though the TOGAF knowledge is thing I value most.

Although it has been 3 or 4 years since we’ve developed a Data Warehouse, recently some questions came up on how Data Marts and ODS (operational data stores) may fit into the architecture.  It was good to have a refresher discussion…

Data Marts and Data Warehouses serve to collect and conform (clean and massage the data so as to support relationships) data from one or many different sources. Besides the ability to access disparate data in a single source, a historical capability is usually added via a technique called ‘Slowly Changing Dimensions’. This allows you to link facts (sales, incidents, change orders…) with dimensions (Stores, Employees, Offices) while preserving the state of the dimension at the time of the fact. For example, when linking a Salesperson to a Territory, if the Salesperson changes on April 1st, the Territory dimension will have 2 records-one that is active through March showing the original Salesperson, and a new record showing the new Salesperson. All related facts that happened prior to April 1 are related to the first record, and so on.

Commonly, the definition of a Data Mart is a subset or specialization of a Data Warehouse. The Data Mart is intended to meet local needs, such as those of a particular department, and, because of the smaller scope and fewer enterprise conformance requirements, Data Marts can be deployed more quickly. Which comes first is more of an academic debate, and depending on the nature of your needs and organization, a single Data Warehouse may suffice.

Data Warehouses have been around a long time and there are generally 2 implementation styles – Kimball and Inmon. Most enterprises follow the Kimball approach, which centers on Dimensions and Facts. If your organization is considering a Data Warehouse, I recommend picking up Kimball’s book: The Data Warehouse Toolkit. If yours is a Microsoft shop, go for the Microsoft specific version. This book gives simple, comprehensive best practices and step-by-step guidance that will produce a solid implementation. 

A TOGAF ADM cycle specifies a remarkable variety of activities for conducting the Enterprise Architecture.  Taking a break from exam prep, I pulled this impressive list of action verbs out of the ADM steps:

activate
approve
articulate
assess
assign
check
conduct
confirm
create
define
deliver
deploy
derive
determine
develop
establish
estimate
evaluate
exploit
finalize
formulate
generate
get
guide
identify
implement
manage
organize
perform
prioritize
refine
request
resolve
review
scope
select
tailor
understand
validate

Summary of today, 12th June 2011:
Today was all about storage. There are a few options in an EC2 instance and it is important to understand the consequences of each. Elastic Block Storage has been very straight-forwrad, but I needed to learn more about instance storage, commonly called ‘ephemeral storage’.

Ephemeral storage, I believe, is disk-based storage on the host (that’s good enough for me anyway). This storage option does not persist across compute stops or terminations but does across reboots (neat). Most compute types are allocated storage capacity ranging from 1.6Gb – 1.7Tb or so, but no drives are mapped to said storage from the default Amazon AMIs, and I don’t think they can be added to a running instance. Ephemeral storage is attractive because it is free (included in price of the server and given the right application makes an attractive option). Not only free storage but also free transfer/IO. Amazon charges nominally for IO requests($.10 per 1 million but that will add up quick for an enterprise) for EBS, as well as storage based on size used.

Some use cases covered…

1) Add ephemeral storage to my instance. Start instance from ami having only root (C:\) with ephemeral drives 0 and 1. You can’t control the drive letter, as far as I know. Also of note: the device names and options differ depending on the server class (micro, small, large, etc.). For example, the t1.micro doesn’t even offer ephemeral storage, while the m1.xlarge (extra large) offers 4 ephemeral drives.

2) Start a new instance, but use existing data. Given the hypothetical server dies but data okay, I am looking at ways of returning to service. So for this use case, I wanted to use an existing volume of data (that was previously attached to a running instance).

First variation was to use a snapshot. I was able to launch a new instance an map the block devices directly to snapshots using Amazon SDK. The snapshot is of a single EBS volume. The default behavior on starting new instances is new EBS volumes with same configuration as image are created. This worked flawlessly and was very straght-forward.

Second variation was to use an existing volume. With a few API call, one can attach existing volumes to stopped instance. If the volume was already attached to another instance, a detach is required or an error is returned.

3) Experience how ephemeral storage works with server reboots, stops and starts, and new images. I launched an instance via Amazon SDK giving it 2 ephemeral drives via Block Device Mappings (this added 2 drives of instance storage to an instance, which was made from an image that had zero ephemeral drives mapped). They worked great, by the way. Then, I stopped and started the instance. Result: the data from the drives (G,H) was gone, but the drive mapping persisted across stops. The documentation says the data will survive a reboot, however. I then created an image from this instance to find out first hand what happens to ephemeral storage. After a few minutes, I was able to launch an instance from this new image and check to see if drives there. It was the same result: the data is gone, but the drive mapping persists across image creation. This was for an instance of the same type running when the image was created. Curious, I launched a micro, which doesn’t offer ephemeral storage. As suspected, the ‘G’ and ‘H’ drives were not there. I suspect it is a similar story for xxl types with 4 drives mapped launched as Large types or others with only 2 available.

4) Determine approach for managing this infrastructure. While the AWS console is a welcome feature, it does not support all of the AWS features. A console alone is not appropriate for managing an infrastructure; there are too many tasks that need to be automated, some requiring dynamic configuration. A powerful infrastructure like Amazon AWS needs a powerful set of command line tools. I am looking at several options here, and there are many routes one could go, such as:

• boto (one-man show, python, read about it in book, not interested-there are tools closer to home)
• Amazon EC2 command line tools. These look rich and full-featured, and they are produced by Amazon, but it requires Java VM and a touch of configuration for Windows. Still attractive, however.
• Powershell to script classes in the Amazon SDK for .Net. I did a quick sample: list buckets for my s3 account and worked the first time-always a good sign.

All this leading up to the big question about managing an enterprise class SQL Server in the cloud. Backup and recovery forthcoming… Some helpful refs on that topic: http://www.brentozar.com/archive/2008/10/running-sql-server-2005-on-amazon-ec2/

http://friism.com/ec2-sql-server-backup-strategies-and-tactics

Up until today, I went on the following assumption regarding a target SharePoint 2010 deployment:

We will maintain 1 image for a WFE, scaling out instances from this single image. This assumed some complicated scripting, on first startup that

1. Set a unique id in the registry (for joining to SP farm)
2. Joined itself to the SharePoint farm, done through command line
3. Installed necessary services other than WFE (e.g., Query), unsure about this

Today we switched gears and disposed of this idea. The alternative, much the same way it is done today, is to build out and maintain the number of images we plan to use. A single instance for the WFE will be launched from its image. At the cost of flex loss and increased administration, we gain some much appreciated simplicity. The SharePoint farm is blissfully ignorant of how many WFEs are behind our load balancer, so this approach supports the scalability we seek.

Once that was settled, I dropped the thread of deploying scripts to the image, dynamically doing some tasks (via instance userdata) and picked up the more robust and promising CloudFormation service. This service allows the specification of each resource, in finite detail and with great flexibility, in an entire Amazon AWS stack. Taking baby steps, I wanted to ensure I could still meet Goal 1: instance management. This took a little learning (of course much more to go), but was readily achievable with a fairly straight-forward template. With the template I created, I am able to:

• Launch a new instance from a specified image id
• Assign an Elastic IP to the new instance
• Get emails about the creation of the instance (this came for free!)
• Set SecurityGroups* and Key Pair
• Set some userdata, though my image doesn’t use this
• Set a few tags for administration

Another interesting feature is that with a single click you can tear down the entire stack. Using CloudFormation (and the AWS SDK for .Net solely to create and delete the stack automated), I have met all objectives for goal 1: instance management.

This brings a new way of thinking for me, confirming what other cloud zealots feel…embracing infrastructure as a template. When something goes awry, launch a new one. This elevated state of mind came when I accepted that failures happen, no matter what. Better to prepare to deal with them than try to prevent them.

*I believe there is a bug with the parameter feature: the type ‘CommaSeparatedList’ is not recognized, though the documentation and examples are straight-forward. This prevents me right now from assigning more than one (existing) security group, as I have to change the type to String. Awaiting answer in AWS forums. Not a show stopper though, because security groups can be created via the stack template, it’s just perhaps more to manage.

Yesterday: sized up needs for a small spike: deploying a small farm SharePoint 2010 to Amazon EC2. To save money, we chose 3 Large compute instances (Web Front end, SQL Server, and Services). We will automanage the termination and launch of instances so we are only paying for about 9 hours/day.

Today: installation and configuration of SharePoint and SQL Server. We decided to use an Active-Directory domain configuration, so we launched a micro instance and setup/configured Active Directory domain services there. We named our domain ‘cloud.com’.

We set some objectives for our goal of learning auto management in AWS. The goals are built around the hellocloud.aspx application.

Objectives:

1. Part 1: Instance
   1. Registers self with a static endpoint (such as an elastic ip, elastic load balancer, or route 53)
   2. Sends email to us when entering service
   3. Can be torn down and restarted each hour automatically
2. Part 2: Backup
   1. Volumes mapped as drives can be reattached automatically
   2. Volumes are backed up to s3
   3. an s3 backup volume is restored
   4. This is done with a script or semi-automated way
3. Part 3: Reliability/availability
   1. Application auto scales and manual scales (out)
   2. Application uses elastic load balancing
   3. We get advanced monitoring
4. Part 4: SQL Server Recoverability
   1. TBD: we decided to get more experience before we started asking questions