“Let’s buy new & get less!”​ said no one ever, yet …

Everyone reading this article is a consumer and as consumers when you bought your last vehicle did you set out to purposely buy that vehicle for it to underperform your current vehicle? Less fuel economy decreased reliability and slower acceleration. Likewise, when looking at different manufacturers and models of the same model year, do you seek the model that underperforms?

No, this isn’t what any of us expect. If you could buy a vehicle that was more reliable, had better fuel economy and faster over another of the same model year, that is very compelling.

Benchmark Observations

As I look over the SAP SD benchmark results for HPE models ranging from 2 sockets, 4 sockets, 8 sockets and 16 sockets I can’t help but make some observations. Likewise, when looking at IBM POWER9 results, some of which are unpublished I can’t help but also make some observations.

Tale of Two Platforms

General observations are this. HPE Intel systems deliver less performance on systems with more sockets. It ranges from just over 2% to over 40% decrease in per-core performance. Regarding POWER9, per-core performance increases going from 2 to 4 sockets by 2.5%, is essentially flat between 4 and 8 sockets and sees a 4% decrease in per-core performance from the 8 socket to the 16 socket model.

Looking only at the Intel results, there is obvious pressure on the scalability for their servers as they scale from 2, 4, 8 and 16 sockets. This isn’t unique to HPE by the way as every Intel vendors results which I looked at exhibited similar results, though my analysis was not exhaustive. Then, if we look at the POWER9 results, they show near-linear scaling regardless of the number of sockets.

Oh My!

The last observation is what I call the ‘foot-race’ results. POWER9 outperforms the other HANA alternatives, anywhere from 220% to 356% per core on systems with the same # of sockets.

S is for Scaling

Of course, POWER9 cores offer SMT8, which is 8 threads per core of parallel and simultaneous execution. SMT is configurable by VM and not the entire server as is the case with alternative systems. For POWER9 VM’s using 96 cores or less, SMT8 is supported. For POWER9 VM’s with more than 96 cores, SAP requires SMT-4 be used. (SAP Note: 2188482). This means a 2 socket 24 core POWER9 server or VM would deliver 192 threads while a 96 core VM would deliver 768 threads. SAP HANA loves threads. The more available, the greater the potential for increased throughput and performance.

Fill in the blank “_orrible”

Regarding Intel systems and threading. Intel’s threading technology is called hyperthreading. It only provides 2 threads per core if enabled. When disabled, there is 1 thread per core. SAP Notes 2100040 & 2711650 state hyperthreading should be disabled for systems with more than 8 sockets. Regarding any size system and hyperthreading, SAP’s position is to go with what each system manufacturer suggests. They have an opinion and recommendation on virtually every aspect of the SAP platform but with this one hot-button issue, they are choosing to pass the buck. For systems using virtualization on Intel systems. VMware in SAP Note 2393917 and VMware KB 55806, they recommend hyperthreading be disabled where the number of vCPU’s match the number of physical cores in the server. Red Hat Virtualization 4.2 states explicitly that hyper-threading must be disabled in SAP Note 2852117. It goes on to say that you should increase the capacity or sizing of the server to compensate for the loss of disabling hyperthreading. Maybe this bumps you from 4 to 8 sockets, who knows.

Is doubling the server a good thing?

This means a 2 socket 56 core server, running bare-metal would deliver 112 threads or if you chose to disable hyperthreading due to the many security vulnerabilities, there would only be 56 threads. Thus, to achieve 112 threads, it would now require a 4 socket server with 112 cores … it’s easy now to see how the project costs could escalate because of these many limitations and restrictions. A 4-socket server with 112 cores and hyperthreading enabled delivers 224 threads, just a few more than the 24 core POWER9. Interesting it would take 112 Intel cores to offer slightly more threads than a 24 core system. Mind-boggling. Continuing on, an 8 socket server with 224 cores offers 448 threads, slightly more than half offered by 96 POWER9 cores. It is pretty clear, the loss of hyperthreading due to the seemingly neverending discovery of new security vulnerabilities with Intel processors appears it will get worse before it gets better.

Summary

As stated, these are just my observations of the two platforms. Other than the obvious per core performance differences, the intent of this article is to provide the reader with information allowing for responsible business decisions. Because who intentionally sets out to buy an underperforming, less reliable, more costly vehicle? As a consumer spending my own money I sure do not and when I am charged with spending my companies money, I also want to be responsible as if I were spending my own.

To learn how to get the most benefit from technology to support your enterprise workloads such as SAP HANA and about Clear Technologies, please visit Clear Technologies SAP Practice. To learn more about our 3 core businesses: Clear Technologies, Visual Storage Intelligence, and our AI practice; Clear Intelligence.

SAP HANA on … #ChooseRight

Window of Opportunity

IBM Power systems started late in the SAP HANA market, on standby if you will when IBM still had their System X business. Once they sold off their x86 business, it opened the door for IBM to work with SAP to offer clients a 2nd platform choice especially with many ECC shops coming from Enterprise platforms and now their only option is to deploy their most critical business application on Intel.

With thousands of clients running SAP ECC using Oracle or DB2 on AIX or running IBM I, there is a large and experienced install base. IBM’s move to support Linux little endian natively beginning with POWER8 eased any development concerns SAP may have had. IBM Power has been fastest adoption for a platform after the initial SAP Ramp-up Program.

Rapid Growth

After the initial Ramp-up program, SAP announced the first GA of HANA for IBM Power in 2015. Since then, it has been the fastest adopted platform by clients to run SAP HANA.

Whether deploying a Greenfield or Brownfield SAP HANA solution, what makes IBM Power such a better platform for SAP HANA over Intel based systems? It starts with its DNA. IBM Power was born an Enterprise system, in the data center running mission critical workloads. Read the Forrester Total Economic Impact of IBM Power Systems for SAP HANA study how they rate the platform.

Flexibility, Performance & Resiliency

• SAP Certified HANA Prod OLTP – 24 TB Scale-up
• SAP Certified HANA Prod OLAP – 24 TB Scale-up
• SAP Certified HANA Prod OLAP – 24 TB Scale-out
• SAP exceptions available upon request
• Up to 16 Production VM’s on E950 & E980
• POWER9 servers can scale up to 64 TB of Memory
• Highly resilient memory offering DDDC+1+1
• Memory sparing, spare chips, ChipKill
• HANA is always virtualized using the integrated IBM PowerVM hypervisor
• Live Partition Mobility
• Dynamically add / remove cores and memory
• Supports TDI 5 delivering greater SAPS per core, up to 2X+
• Offers Elastic Capacity on Demand activations of cores & memory
• Highest Reliability excluding Z for 11 years per ITIC
• Concurrent maintenance features for firmware, drives, PCIe adapters, fans and power supplies
• Concurrent maintenance for the I/O path from VM to SAN and network when using Dual Virtual I/O Servers
• Dynamic tuning & optimization
• Supports SAP Native Storage Extension and Fast Restart
• IBM Storwize storage is optimized for SAP HANA on POWER

Additional features to be announced any day (Nov 5, 2019 is todays timestamp).

• Persistent Memory at no additional cost
• Persistent Memory with no performance degradation
• Use of Shared Processor Pools for Production in addition to existing support for non-Prod
• RHEL 8

Clients will deploy fewer systems, while able to host more workloads per system, whether those are legacy SAP ECC, SolMan or non-SAP workloads such as legacy Oracle workloads or possibly new Cognitive workloads.

Bringing it ALL together

SAP HANA; whether Suite or BW on HANA or S/4HANA or BW/4HANA, businesses tend to focus on the application, discounting the infrastructure as commodity – it’s all the same. With SAP HANA, designed as a scale-up in-memory technology, IBM Power is the optimal platform to host it.

• Primary benefits such as fewer systems with greater utilization.

• Secondary benefits such as less infrastructure and data center services required, i.e. fewer network & SAN ports, fewer power plugs with lower electrical consumption requiring less to cool.

• Tertiary benefits, often more difficult to quantify such as the downtime the business did NOT have to take to perform a maintenance action such as updating firmware or adding an adapter for additional capacity.
• Other actions such as downstream activities impacting the I/O paths like a network switch service event can all be accommodated with a properly architected and deployed Power solution.

These foundational capabilities allow the business to remain on schedule, consultants continue to work and not be idle.

#ChooseRight

There are only two options for SAP HANA. One option is the platform forcing you to choose one feature for another making every decision a compromise. The other option is the platform offering complete flexibility, scalability and resiliency with no compromises as even IDC states in this whitepaper. No one wants to go back to their board asking for more money admitting they made a mistake, undersized or failed to anticipate something, so #ChooseRight!

HANA – Winning with IBM POWER

The Business Challenge: A $4B manufacturing company decided to migrate their AnyDB to Suite on HANA; but on which platform? 

The Competition: The client evaluated a TDI solution from an Intel based vendor + major storage vendor as part of a converged infrastructure solution as well as a TDI solution from IBM Power + IBM Storage. The SAP Basis Manager favored an Intel solution; either in the cloud or on-premises based on the perception is was the lowest cost, optimal platform for HANA with comparable features to the other choice.

The Evaluation: I was the Client Executive and Executive Architect for my team, leading the design and competitive effort. The clients initial objection to IBM Power stemmed from their view of the current platform running SAP ECC. Though it had done so, with virtually no issues for 20 years, they viewed it as expensive, inflexible and legacy. This was largely due to the fact they had not implemented all of the virtualization capabilities, having allowed the system to grow with dedicated resources. Also, due to some in-house mistakes with the current storage, their answer was to buy more hardware vs tightening up their own internal procedures plus key individuals taking ownership for the mistake which led to the problem.

The clients SAP team thought this would be a simple exercise. Get the Intel solution price and the IBM Power solution price, present to management for the rubber stamp to move off to the next phase, migration prep. Only there was a problem. the Intel solution costing was coming in higher than expected. The AnyDB size of 24 TB reduced down to ~4.1 TB, even to 3.6 TB with cleanup, per the SAP Quicksizer. True story – the Intel team proposed they start with 3 TB systems to get them on the floor, then wait and see if they truly needed more. Thankfully, the client didn’t accept this generous offer, requesting they quote 6 TB systems as that was the next increment. With IBM Power, you can configure DIMMs to more closely match the required capacity as the platform does not have the memory placement requirements (and limitations) as Intel platforms do. This client further required all environments be sized for the full HANA DB copy, which had grown from 4.1 TB to 5.1 TB (plus OS taking it to 5.3 TB). Good thing they didn’t go with those 3 TB Intel systems, eh?

The Intel solutions configured every HANA DB environment as bare-metal because the memory requirements didn’t support VMware for virtualization. It wasn’t an option due to the VMware maximum memory requirement was less than the client required. The memory sizing also pushed the Intel solution into larger servers with more sockets. The environments consisted of Sandbox, Dev, QA and Production. Each had a full memory sized HANA DB on bare-metal servers while they did use VMware to host the NetWeaver application servers; though again, they had multiple ones for each environment. I don’t know the exact numbers but believe they were north of 16 Intel servers with 10 of those configured as 6 TB bare-metal servers.

The IBM Power solution consisted of 3 x POWER9 servers. Yes, 3. Everything was fully virtualized, designed for maximum resiliency and serviceability. 2 smaller POWER9 servers for Production, each hosting 1 HANA DB + 2 NetWeaver VM’s for Production. DR hosted a server with 3X the capacity of the Production server. This single, highly performant and reliable server was configured for 17 VM’s. Sandbox, Dev and QA each had 1 VM for HANA DB and 2 VM’s for their NetWeaver App servers. Prod had 1 VM and 2 NetWeaver App servers for failover plus 2 VM’s for the Dual Virtual I/O Servers (VIOS). At each site, the servers were connected to an IBM flash storage solution using redundant IBM branded SAN switches.

The Decision: Management was presented with both options. Feedback was given to the SAP team. My team didn’t do anything but wait as the word was the Intel team was scrambling to “fix” their numbers. Updated pricing was presented to management and a decision was made. They chose the IBM Power solution running SUSE Linux with the IBM Storwize all flash storage for their Suite on HANA solution. Their justification was simple. The incredible reliability and performance record of the existing IBM Power spoke for itself – they had actual experience running SAP on it, albeit not HANA but who was I to split hairs. Secondly, and probably most important, the IBM Power solution was at least 35% less costly than the Intel solution. By the way, I had submitted a proposal for the migration services as well. Going up against a couple of big players. I won … beat them by 30% as well.

In Conclusion: though management didn’t know at the time, or at least couldn’t fully comprehend the benefits they would obtain with the virtualization capabilities which come with IBM Power servers that are #NoCompromise as this would play a key role during the 6 month migration window allowing their consultants and business leaders flexibility to provision, change, update, modify (you name it) and more the many requests which came up suddenly without downtime, added cost or delays.

I oversaw the implementation and migration effort, which started by ensuring the solution was properly designed with all of its pieces, ordered and the client environment prepared. Then, working closely with the migration team, ensuring we understood each others roles, not just our but the platforms capabilities as well as the current and future timelines. Took this all the way to the final Go-Live migration which went off like clockwork. Down at 10 pm Friday night. Migration done by 7 am Sunday morning. Clean-up and other details tended to to Go-Live by 4 pm Sunday afternoon. And they haven’t had to take an outage since … at least not for anything hardware related.

Reach out if you want to learn how my team designs world class systems, using world class assessment tools and migration techniques which allow our solutions to be optimized, faster, efficient and ultimately lower cost.

Another Successful Go-Live for SAP HANA on POWER!

My client is now live, running Suite on HANA 2.0 on an IBM POWER & Storwize solution after a successful weekend migration.

The focus for this blog is to discuss what led to my client successfully migrating their SAP ECC environment to SAP HANA using capabilities inherent with IBM Power servers.  Capabilities available to every SAP HANA client who choose IBM Power vs the only other alternative platform supported for SAP HANA workloads.  The alternative option is built with Intel processors either running as bare-metal or with virtualization.  If virtualized, it will likely be VMware which I refer to as a “compromise” solution full of gotchas, limitations, restrictions and constraints.  Whereas,  IBM Power using PowerVM  is a “No Compromise” option.   I’ll give some examples of this bold statement below.

Back to my client.  In the Fall of 2018, the client chose an IBM Power solution  supporting four environments using a fully virtualized two-site solution.  Client chose to deploy HANA in parallel to their existing SAP ECC environment which runs on IBM i.   Regarding storage, each site uses  new IBM Storwize AFA products.  For high-availability and resiliency, the solution uses SuSE HA clustering between a pair of Production servers with SAP HANA Replication locally from Primary Prod to Failover Prod and then  from Primary Prod to the DR server.

DR consists of a (very) large Scale-up IBM POWER server hosting all HANA DB & App (NetWeaver) VM’s for each environment (Sandbox, Dev, QAS, etc).  Production uses a smaller pair of the DR server just for the HANA DB VM’s plus VM’s hosting redundant App servers. 

Each of the IBM Power systems in this SAP HANA environment use Dual VIOS or Virtual I/O Server.  For the uninitiated, Dual VIOS means there are 2 special VM’s which virtualize and manage the I/O to every VM’s.  Remember a VM can technically use any combination of dedicated or virtual I/O but typically when a VIOS is used, it manages both network and storage I/O with one exception where I often see a client use a dedicated Fibre adapter for physical tape connections.  Benefits of implementing Dual VIOS are many. They require fewer adapters leading to smaller servers and/or less I/O expansion, provide I/O path redundancy to network & storage while also increasing serviceability as the client can do just about any kind of maintenance on the I/O path transparent to the workloads.  This means very little downtime is ever required to service and maintain the I/O subsystem.  This includes adding, removing,  upgrading and configuring adapters,  ports, updating drivers, etc.  Thus, if a port or adapter fails or if something were to happen to a VIOS (very rare), its redundant ports, adapters and VIOS are configured to automatically service the I/O from the remaining resources.   There are many options available to deploy redundant VIOS, from active/passive to active/active for both network and storage I/O.  Another benefit when virtualizing the I/O is to enable features such as Live Partition Mobility and Simplified Remote Restart ….. no compromises, remember?!

I should disclose that my company has an SAP migration, consulting and managed services practice.  We were selected to provide both the infrastructure implementation and the SAP ECC to HANA migration services.  Starting with the lower environments, my SAP services team started late last year (2018) concluding with Prod in May 2019.  This client wanted each environment to be a full copy of the HANA DB whereas it is common for clients to make the lower environments smaller. Our migration and infrastructure teams worked together during every step, creating additional VM’s, adding storage, mount points, dialing in  cores and memory for every HANA DB and App VM for each environment. 

With IBM POWER9 servers, SAP states Production VM’s are required to use Dedicated (and dedicated donating) cores while non-Prod environments may use dedicated cores or use Shared Processor Pools (SPP).  This means clients can use every square inch of their IBM POWER servers – dialing in the cores and memory.  For Non-Prod, clients receive greater granularity sharing cores leading to even greater resource efficiency.  This leads to smaller and fewer servers – say it with me “lower cost!” make for very happy clients!

Contrast this with the alternative Intel solution with its two choices; bare-metal meaning no virtualization benefits or to use virtualization.  Bare-metal means 1 OS image per physical server. Hopefully your infrastructure provider or SAP consultant does not under size the cores & memory as the cost to remediate can be very costly (i.e. possibly new servers if the current system is already maxed out).  If the market leading virtualization product (i.e. VMware) is chosen, its VMs do not offer the granularity available as are available from IBM Power systems with its ultra-secure and rock-solid Power Hypervisor (PHYP).

The alternative virtualization product requires (i.e. limits or restricts) each VM to allocate cores in increments of full or ½ sockets.  Let’s say the HANA DB system is a 4-socket Intel server using 22-core processors totaling 88 cores with 1536 GB RAM per socket or 6 TB in total.  If the HANA DB sizing called for 46 cores, you would be required to assign 3 sockets or 66 cores for a VM which only needs 46, wasting 20 cores plus all of the excess memory attached to that 3^rd socket.  Another approach using this example of waste when using virtualization for the alternative option.   If the HANA DB VM requires 3200 GB of memory.  Because this is 128 GB more than physically connected to 2 sockets, you must allocate all 4,608 GB of memory attached to the 3 sockets as well as all 66 cores on those 3 sockets as previously described.  1,408 GB of memory is wasted, unable to be used by any other VM’s on that server.  Fortunately, the larger DIMMs used to achieve the needed capacities are cheap so this waste is a drop in the bucket (in reality, these large DIMMs are NOT cheap at all!)  SAP states there is overhead incurred from this market leading virtualization product.  Also, if security is important, don’t overlook its many security vulnerabilities such as  the Intel Management Engine, VMware vSphere, Linux plus the recent Meltdown, Spectre, Foreshadow and Zombieload  side-channel threats come to mind.

Some of these security vulnerabilities come with a performance penalty.  SAP fully supports IBM POWER8 & POWER9 using SMT8 while many recommend disabling the use of hyper-threading when using Intel servers.  SMT options are SMT8, SMT4, SMT2 and SMT0 (ie ST0). To view the SMT level on SuSE or RedHat, use `ppc64_cpu –smt` and to change the SMT level from its current level to SMT4, for example, use `ppc64_cpu –smt=4`.  Note the switch used is a “<dash><dash>smt” as many editors will change that to a large single dash.  The default for Linux on POWER8 should be SMT8 but there are some situations where the default is SMT4.  For POWER9, all supported Linux distributions should default to SMT8.  Also, clients are able to change SMT from one level to another dynamically by VM (yes, I said by “VM”).  This is a huge feature,  unavailable on Intel.

UPDATE (6/18/2019): SAP Note 2393917 states the following statement. “Due to the security vulnerability identified in CVE-2018-3646 VMware strongly advices customers to review and enable the recommendations indicated in VMware KB 55806. In particular, VMWare recommends that customers must ensure that after enablement, the maximum number of vCPUs per VM must be less than or equal to the number of total cores available on the system or the VM will fail to power on. The number of vCPUs on the VM may need to be reduced if it is to run on existing hardware. The number of vCPUs should be a factor of two. VMware is providing a tool to assist customers with the analysis of their VM configuration.” .  This SAP Note is very explicit. Though they are not declaring clients MUST disable hyper-threading, when a vendor states they “strongly advise” you to do something, they are saying essentially telling you to do something.

Here are a couple of articles  on the performance impact here and here, but do your own internet research as well.  SAP has been remediating their cloud Intel environment with details in SAP Note 2709955 but are copping out regarding what clients should do regarding their on-premise Intel servers.  Instead, they defer to the relevant vendors like Intel, VMware, RedHat, SuSE, etc to determine what they should do.  It’s not like hyper-threading is known for performance or  throughput, for that matter but if it delivers something to increase efficiency, it would be a good thing.  Lose hyper-threading and all you have left for threads are physical cores. This means less efficiency for the application as HANA loves threads which is why it scales so well on IBM Power.  For Intel sizing, this likely requires more cores with associated memory which leads to more sockets with more memory leading to larger, more expensive servers to obtain the desired scale.  Using TDI Phase 5  based on SAPS values, any sizing would need to be adjusted to compensate for not having hyper-threading.    With IBM Power, size it, dial it in the cores and memory, tune the OS and re-use spare capacity for other VM’s running Linux, AIX & IBM i (if supported by the chosen model) as needed – no compromises!

Thought I would create the table below to compare IBM’s PowerVM vs VMware’s vSphere using a couple of SAP Notes to compare PowerVM versus VMware. It gets really complicated trying to explain Intel and VMware capabilities, even IBM Power & PowerVM to a lesser degree as what is supported varies by CPU Architecture (i.e. Ivy Bridge, Haswell, Broadwell, Skylake and Cascade Lake in the case of Intel and POWER8 vs POWER9) as well as the VMware generation (i.e. vSphere pre-6.5, 6.5 and 6.7).  I’ll try to annotate differences but will ask you to reference SAP Notes for VMware vSphere on Intel are 2652670, 2718982 and 2393917 for specific details.  SAP Notes for IBM Power using PowerVM are 2055470,  2230704, 2188482 and 2535891

Disclaimer: I am verifying some of the values shown in the table below and will update the table as needed.  With new features being supported regularly, it can be a challenge to remain current.

 

REMINDER: This chart ONLY applies to Business Suite and not BW. I’d have to build another table for those differences as BW was not germane to this client or this blog.

	VMware vSphere (OLTP)	PowerVM (OLTP)
Max VM’s per system	16**	1 – 16 Production VM’s*   1 – 1008 VM’s (15 Prod + 993 Non-Prod VM’s) POWER9 E950 & E980
		1 – 8 Production VM’s*   1 – 1000 VM’s (7 Prod + 993    Non-Prod VM’s) POWER8 E870(C) & E880(C)
		1 – 6 Production VM’s*   1 – 920 VM’s (5 Prod + 915      Non-Prod VM’s) POWER8 E850C
		1 – 4 Production VM’s*   1 – 426 VM’s (3 Prod + 423    Non-Prod VM’s POWER8 & POWER9 2-socket Scale-Out
Max VM size	Up to 4 sockets (BW, SL, CL)
VM size increments***	1, 2, 3 and 4 full sockets	Dedicated & Dedicated Donating   1 core increments
	½ socket (no multiples like 1.5) but 2, 3, 4, 5, 6, 8 and 8 ½ sockets supported	Shared Processor Pool (Non-Prod workloads)   Rule of Thumb is 20 VM’s per core
Threading	2 threads / Hyper-Thread	SMT8 per core or virtual processor
Max vCPU	128 (6.5, 6.7)(BW)	POWER8: If a VM uses more than 96 cores then set SMT=4. Otherwise set SMT=8 Max Cores per VM * SMT level = threads Ex 1: 176*4=704 threads Ex 2: VM1 = 96 *8 = 768 threads and VM2 = 80*8 = 640 threads for a total of 1,408 threads.
	192 (6.7)(BW)	POWER9: If a VM uses more than 48 cores then set SMT=4. Otherwise set SMT=8. Max Cores per VM * SMT level = threads Ex 1: 128*4=512 threads and 64*4= 256 threads for a total of 768 threads. Ex 2: VM1 = 48 *8 = 384 threads; VM2 = 48*8 = 384; VM3 = 48*8 = 384; VM4 = 48*8 = 384 threads for a total of 1,536 threads.
	128 (6.5, 6.7)(SL,CL)	Cores * Max # of vCPU per core * SMT level Using SPP, take the # of cores * 20 * SMT level. POWER8 Ex: 176 * 20 * 8 = 28,160 threads POWER9 Ex: 192 * 20 * 8 = 30,720 threads
	224 (6.5, 6.7)(SL,CL)
Max Cores	N/A	176 cores (POWER8)
	N/A	192 cores (POWER9)
Max Memory	4 TB (6.5, 6.7)(BW)	16 TB (POWER8)
	6 TB (6.5, 6.7)(SL, CL)	24 TB (POWER9)
Memory allocation	Only memory attached to ½ socket or full sockets. If more memory is required, underlying ½ or full sockets go with it.	As long as the VM has the minimum memory allocated, memory increments can occur as small as 1 MB
SAP could require virtualization to reproduce on bare-metal	Yes	No
Min performance degradation from virtualization per SAP	14% for ½ socket VM’s   Avg of 10% over bare-metal	0%

* VIOS VM’s do not count toward these totals

** Requires an 8-socket server to achieve 16 VM’s, with each using ½ sockets per VM. If using full socket VM’s the most possible would be 8 using the 8-socket server example.

*** You can mix ½ and full socket VM’s on the same server. Example would be 4 x ½ socket VM’s which would consume 2 sockets and 6 x 1 socket VMs consuming 6 sockets totaling 8 sockets.

BW = Broadwell

SL = Sky Lake

CL = Cannon Lake

Back to the migration story.  It is difficult to coordinate downtime among the various stakeholders of a multi-billion-dollar company not to mention the cost of downtime.  Since they were migrating the database from the current ECC system over the network, the client liked having the option to granularly allocate resources plus move VM’s where they needed them.   With IBM Power, clients have flexibility leading to fewer scheduled outages as most maintenance and administration can be performed concurrently.  Is anyone keeping score of all the advantages obtained by IBM Power as I’ve put many hash marks in its column while placing many X’s in the column for the alternative platform.

Regarding the network traffic, the network adapters are 10 GbE optical, configured in the VIOS using Shared Ethernet Adapters which provide a virtual switch. Traffic enters and leaves the server through the SEA whereby network packets within the server are sent/received over the systems memory bus using a technology in the Power Hypervisor called Virtual Ethernet (VE).   This makes data transfers from VM to VM within the frame occur very fast with ultra-low latency and efficiency.  Hence why the client prefers the App servers to reside (logically and physically) millimeters away from the HANA DB server.

The export of the 24 TB database from the source system began just after midnight Friday night. This took approximately 6 hours.  They next moved to import the data into the new environment which took 24 hours.   During the migration, the client chose to stuff more cores and memory to the App VM while running on the Scale-up server.  The App VM was originally sized to use 4 cores and 64 GB RAM but they called an audible and bumped the cores to 12 cores and memory to 384 GB RAM.  For those familiar or not with Power systems and common workloads, this is a lot of cores and memory for an App server, but since they had spare resources on the Scale-up server, they chose to “use ‘em since they got ‘em!”.  After the migration, the App VM’s were reduced from their inflated values to the go-forward (lower) values reducing the cores 1 at a time and the memory 16 GB at a time, performing this dynamically.  Though dynamic add/remove of cores and memory is technically supported by SuSE and RedHat running in IBM Power, SAP doesn’t yet support it on either IBM Power or Intel, I can verify this can be successfully performed when the changes are made in these small increments vs one action.  

The Client has been working through their post-migration punch-list as the system went live on schedule at 4:30 pm that Sunday afternoon.  Starting with a kick-off call Friday night at 8:30 pm and going live Sunday at 4:30 pm they successfully moved their entire businesses Production environment from SAP ECC to Suite on HANA in 44 hours (31 hours for the start of export to finish of import).

Beginning last Fall, my team began to implement the infrastructure starting with the DR environment. Over the many months, we received many requests from the SAP Basis team and our SAP migration team to create new environments for testing, add resources, mount points or make some type of change to the VM.  The only feature which would’ve been beneficial to have and  still unsupported by SAP on either supported Platform is the ability to dynamically add/remove cores and memory.  I do expect this feature to be supported on IBM Power with PowerVM shortly.  These capabilities, especially dynamic memory add / removal have been around for a decade and a half with IBM Power.  Technology is very reliable, very consistent and very convenient.  I’m sure purists for Intel solutions using VMware might argue their product works just as well.  I believe SAP’s own guidance says otherwise and of course if someone would like to have some fun, we could setup a 2-server solution and run through a battery of tests to compare virtualization features on both platforms.  We’d have to run each under a heavy load as it would be unfair to our audience to do these tests in a vacuum as that isn’t real world.  While at it, maybe we could run some informal Oracle database (sorry, can’t help myself – read my previous blog to know of my Oracle obsession) testing along with a TCA/TCO analysis comparing how both platforms performs. We’ll refer to it as “using a leading enterprise RDBMS product” so we don’t upset lawyers.

In summary, I’m obviously very proud how this solution performed as it took a strong, capable team to design, deploy and support this client for 4 separate migrations.  This no-compromise solution was >35% less costly vs a competing solution making the lives of this client much better from beginning to go-live.

Kudo’s to IBM as they have  the best platform for SAP HANA and also  tremendous SAP talent available to partners and clients for pre-sales support, IBM Lab Services for HANA installation assistance and IBM Linux for SAP HANA support.

 

Excellent Resources:

IBM Systems Magazine http://ibmsystemsmag.com/power/systems-management/data-management/sap-hana-landscapes/?utm_source=SilverpopMailing&utm_medium=email&utm_campaign=052119-Power-EXTRA+%281%29+Live+Send&utm_content=Simplify+and+Accelerate+SAP+HANA+Landscapes&spMailingID=15684623&spUserID=MTMzMTk5NTQyNjAxS0&spJobID=1641419654&spReportId=MTY0MTQxOTY1NAS2#.XOVt36iYHsI.twitter

 

SAP on Power blog by Alfred Freudenberger https://saponpower.wordpress.com

 

Linux on Power – system tuning Linux https://developer.ibm.com/linuxonpower/docs/linux-on-power-system-tuning/

 

Interesting article discussing the use of SMT8 on IBM POWER9 servers running DB2 https://developer.ibm.com/linuxonpower/2018/04/19/ibm-power9-smt-performance-db2/

SAP HANA – could I have extra complexity please?

Just returned from IBM’s Systems Technical University conference held in Orlando having delivered presentations on 4 different topics.

1. Benefits of SAP HANA on POWER vs Intel
2. Why IBM POWER systems are datacenter leaders
3. Only platform that controls Software Licensing
4. Why DB2 beats Oracle on POWER (implied that it beats Intel).

With the SAP Sapphire conference last week in Orlando, there was a slew of announcements.  Quick reminder for the uninitiated with SAP HANA, that it is ONLY supported on Intel and POWER based systems running one OS; SUSE or RedHat Linux. With that, IBM POWER continues to deliver the best value.

What is the value offered with the POWER stack? Flexibility! It really is that simple.  If I had a mic on the plane as I write this, I would drop it. Conversely, what is the value offered going with an Intel stack? Compromise!

Some of the flexibility offered thru IBM POWER systems are: Scale-up, scale-out, complete virtualization, grow, shrink, move, perform concurrent maintenance, mix workloads: existing ECC workloads on AIX or IBM i with new HANA running Linux all on the same server.  All of this runs using the most resilient HANA platform available.

Why do I label Intel systems as “Compromise” solutions? It isn’t a competitive shot nor FUD.  Listen, as an Client Executive and Executive Architect for an Channel Reseller, I am able to offer my clients solutions from multiple vendors that include IBM POWER and Intel based systems manufacturers.  I’ve made the conscious decision though to promote IBM POWER over Intel.  Why? Because I not only believe in the capabilities of the platform but also having worked with some of the largest companies in the world, I regularly hear and see the impact running Enterprise workloads on Intel based servers has on the business.

If you read my previous blog, I mention a client who just recently moved their Oracle workloads from POWER to Intel.  Within months, they’ve had to buy over $5M in new licenses going from a simple standalone and a few 2-node clusters (all on the same servers) to an 8-node VMware based Oracle RAC cluster.  This environment is having daily stability issues significantly impacting their business.  Yes, their decision to standardize on a single platform has introduced complexity to the business costing them money, resources (exhausted & not having the proper skills to manage the complexity) that impacts their end-users.

The “Compromise” I mention to host SAP HANA on Intel is that everything has to be an asterisk by it – in other words a limitation or restriction – everything requires follow-up questions and research to ensure what the business wants to do, can be done. Here are some examples.
1) VMware vSphere 5.5 initially supported 1 VM per system which has now been increased to 4 VM’s, but with many qualifications.
a) Restricted to 2 & 4 socket Intel servers
1) VM’s are limited to a socket
2) 2 socket server ONLY supports 2 VM’s, 4 socket would be 4 x 1 sockets each
b) Only E5_v2, E5_v3, E7_v2 and E7_v3 chips are supported – NO Broadwell
c) Want to redeploy capacity for other? Appliances certified only for SoH or S4H
uses cannot be used for other purposes such as BW
d) Did I mention, those VM’s are also limited to 64 vCPU and 1 TB of memory each
e) If a VM needs more memory than what is attached to that socket? No problem, you have to add an additional socket and all of its memory – no sharing!
2) VMware vSphere 6.0 just recently went from 1 to 16 VM’s per system.
a) VM’s are still limited to a socket or 1/2 socket.
b) 1/2 socket isn’t as amazing as it sounds.  Since vSphere supports 2, 4 & 8 socket servers, there can be 16 x 1/2 socket VM’s.
c) What there cannot be, is any combination of VM’s >1 socket with 1/2 socket assigned. In other words, a VM cannot have 1.5 or 3.5 sockets. Any VM resource requirement above 1 socket requires the addition of an entire socket.  1.5 sockets would be 2 sockets.
d) Multi-node setups are NOT permitted …. at all!
e) VM’s larger than 2 sockets cannot use Ivy Bridge based systems, only Haswell or Broadwell chips – but ONLY on 4-socket servers.  Oh my gosh, this is making my head hurt!
f) If using an 8-socket system, it only supports a single production VM using Haswell ONLY processors.  NOT Ivy Bridge and NOT Broadwell!
g) VM’s are limited to 128 vCPU and 4 TB of memory
3) VMware vSphere 6.5 with SAP HANA SPS 12 only supports Intel Broadwell based systems. What if your HANA Appliance is based on Ivy-Bridge or Haswell processor technology? “Where is that Intel rep’s business card? Guess I’ll have to buy another one since I can’t upgrade these”
a) VM’s using >4 sockets are currently NOT supported with these Broadwell chips
b) Now, it gets better. I hope you are writing this down – For 2 OR 8 socket systems, the maximum VM size is 2 sockets.  Only a 4 socket system supports 1 VM with 4 sockets.
c) Same 1/2 socket restrictions as vSphere 6.0.
d) Servers with >8 sockets do NOT permit the use of VMware
e) If your VM requirements exceed 128 vCPU and 4 TB of memory, you must move it to a bare-metal system ….. Call me – I’ll put you on a POWER system where you can scale-up, scale-out without of this mess

Contrast all of these VMware + Intel limitations, restrictions, liabilities, qualification or simply said “Compromise” systems to the IBM Power System.

POWER8 servers run the POWER Hypervisor called PowerVM.  This Hypervisor and its suite of features deliver flexibility allowing all physical, all virtual and a combination of physical & virtual resource usage on each system. Even where there are VM limits such as 4 on the low-end system, that 4 could really be 423 VM’s.  I’m making a theoretical statement here to prove the point. Let’s use a 2 socket 24 core S824 server.  3 VM’s, each with 1 core (yes, I said core) for production usage and the 4th VM’s is really a Shared Processor Pool with 21 cores.  Those 21 cores support up to 20 VM’s per core or 420 VM’s. Any non-production use is permitted.

Each PowerVM VM supports up to 16 TB of memory and 144 cores.  VM size above 108 cores requires the use of SMT4 whereas <=108 cores permit SMT8.  Thus, 144 cores with SMT4 is 576 vCPU’s or 4.5X what Intel can do with 4X the memory footprint.  By the way, that 108 core VM would support 864 vCPU’s – just saying!  Note: I need to verify as the largest SMT8 VM may be 96 cores with only 768 vCPU.

Not only can we allocate physical cores to VM’s and NOT limited to 1/2 or full socket increments like Intel, but POWER systems granularity allows for adjustments at the vCPU level.

PowerVM supports scale-out and scale-up.  Then again, if you have heard or read about the Pfizer story for scale-out BW, you might rethink a literal scale-out approach. Read IBM’s Alfred Freudenberger’s blog on this subject at https://saponpower.wordpress.com/2016/05/26/update-sap-hana-support-for-vmware-ibm-power-systems-and-new-customer-testimonials/

While on the subject of BWoH/B4H, PowerVM supports 6 TB per VM whereas the vSphere 6.0 supports is 3 TB and the limitations increase from here.

Do you see why I choose to promote IBM Power vs Intel? When I walk into a client, the most valuable item I bring with me is my credibility.  HANA on Intel is a constant train wreck with constant changes & gotcha’s. Clients currently with HANA on Intel solutions or better yet, running ECC on Intel have options.  That option is to move to a HANA 2.0 environment using SUSE 12 or RedHat v7 Linux on POWER servers. Each server will host multiple VM’s with greater resiliency providing the business the flexibility desired from the critical business system that likely touches every part of the business.

Does your IT shop use a combination wrench?

More and more, IT shops seem inclined to consolidate and simplify their infrastructure to one platform. A mindset that all workloads can or should run on a single platform incorporated into ‘Software-defined this’ and ‘Software-defined that’.  It tantalizes the decision makers senses as vendors claim to reduce complexity and cost.

Technology has become Ford vs Chevy or John Deere vs Case International.  Whereas these four vendors each have some unique capabilities and offerings they are also leaders in innovation and reliability.  For IT shops, there is this perception that only Intel & VMware are viable infrastructure options to deploy every workload type.  Mission / Life critical workloads in healthcare, high-frequency financial transactions, HPC, Big Data, Analytics, emerging Cognitive & AI but also traditional ERP workloads that run entire businesses – SAP ECC, SAP HANA and Oracle EBS are probably the most common that I see as there are also some industry specific ones for Industrial and automotive companies – I’m thinking of Infor.

When a new project comes up, there is little thought given to the platform. either the business or maybe the ISV will state what and how many of server X should be ordered. The parts arrive, eventually getting deployed.  Little consideration is given to the total cost of ownership or the the impact to the business caused by the system complexity.

I’ve watched a client move their Oracle workloads to IBM POWER several years ago. This allowed them to reduce their software licensing and annual maintenance cost as well as to redeploy licensing to other projects – cost avoidance by not having to add net new licensing.  As it happens in business, people moved on, out and up. New people came in whose answer to everything was Intel + VMware.  Yes, a combination wrench.

If any of you have used a combination wrench,  you know there are a few times it is the proper tool. However, it can also strip or round over the head of a bolt or nut if too much pressure or torque is applied. Sometimes the proper tool is a SAE or Metric box wrench, possible a socket, even an impact wrench.  In this clients case, they have started to move their Oracle workloads from POWER to Intel.  Workloads currently running on standalone servers or at most using 2-node PowerHA clusters.  Moving these simple (little complexity) Oracle VM’s to 6-node VMware Oracle RAC clusters that have now grown to 8-nodes.  Because we all know that Oracle RAC scales really well (please tell me you picked up on the sarcasm).

I heard from the business earlier this year that they had to buy over $5M of net-new Oracle licensing for this new environment. Because of this unforeseen expense, they are moving other commercial products to open-source since we all know that open-source is “free” to offset the Oracle cost.

Oh, I forgot to mention.  That 8-node VMWare Oracle RAC cluster is crashing virtually every day.  I guess they are putting too much pressure on the combination wrench!

Oracle is a mess & customers pay the price!

Chaos that is Oracle

Clients are rapidly adopting open source technologies in support of purpose-built applications while also shifting portions of on-premises workloads to major Cloud providers like Amazon’s AWS, Microsoft’s Azure and IBM’s SoftLayer.  These changes are sending Oracle’s licensing revenue into the tank forcing them to re-tool … I’m being kind saying it this way.

What do we see  Oracle doing these days?

• Aggressively going after VMware environments who use Oracle Enterprise products for licensing infractions
• Pushing each of their clients toward Oracle’s public cloud
• Drastically changing how Oracle is licensed for Authorized Cloud Environments using Intel servers
• Latest evidence indicates they are set to abandon Solaris and SPARC technology
• On-going staff layoffs as they shift resources, priorities & funding from on-premises to cloud initiatives

VMware environments

I’ve previously discussed for running Oracle on Intel (vs IBM POWER), Intel & VMware have an Oracle problem. This was acknowledged by Chad Sakac, Dell EMC’s President Converged Division in his August 17, 2016 blog in what really amounted to an Open Letter to King Larry Ellison, himself. I doubt most businesses using Oracle with VMware & Intel servers fully understand the financial implications this has to their business.  Allow me to paraphrase the essence of the note “Larry, take your boot off the necks of our people”.

This is a very contentious topic so I’ll not take a position but will try to briefly explain both sides.  Oracle’s position is simple even though it is very complex.  Oracle does not recognize VMware as an approved partitioning (view it as soft partitioning) method to limit Oracle licensing. As such, clients running Oracle in a VMware environment, regardless of how little or much is used, must properly license it for every Intel server under that clients Enterprise (assume vSphere 6+).  They really do go beyond a rational argument IMHO. Since Oracle owns the software and authored the rules they use these subtleties to lean on clients extracting massive profits despite what the contract may say. An example that comes to mind is how Oracle suddenly changed licensing configurations for Oracle Standard Edition and Standard Edition One. They sunset both of these products as of December 31, 2015 replacing both with Standard Edition 2. What can only be described as screwing clients, they halved the number of sockets allowed on a server or in a RAC cluster, limited the number of cpu threads per DB instance while doubling the number of minimum Named User Plus (NUPs). On behalf of Larry, he apologizes to any 4 socket Oracle Standard Edition users but if you don’t convert to a 2 socket configuration (2 sockets for 1 server or 1 socket for 2 servers using RAC) then be prepared to license the server using the Oracle Enterprise Edition licensing model.

The Intel server vendors and VMware have a different interpretation on how Oracle should be licensed.  I’ll boil their position down to using host or cpu affinity rules.  House of Bricks published a paper that does a good job trying to defend Intel+VMware’s licensing position. In their effort, they do show how fragile of ground they sit on with its approach  highlighting the risks businesses take if they hitch their wagons to HoB, VMware & at least Dell’s recommenations.

This picture, which I believe House of Bricks gets the credit for creating captures the Oracle licensing model for Intel+VMware environments quite well. When you pull your car into a parking garage – you expect to pay for 1 spot yet Oracle says you must pay for every one as you could technically park in any of them. VMware asserts you should only pay for a single floor at most because your vehicle may not be a compact car, may not have the clearance for all levels, there are reserved & handicapped spots which you can’t use. You get the idea.

It simply a disaster for any business to run Oracle on Intel servers. Oracle wins if you do not virtualize, running each on standalone servers.  Oracle wins if you use VMware, regardless of how little or much you actually us.  Be prepared to pay or to litigate!

Oracle and the “Cloud”

This topic is more difficult to provide sources so I’ll just stick to anecdotal evidence. Take it or leave it. At contract renewal, adding products to contracts or new projects like migrating JD Edwards “World” to “Enterprise One” or a new Oracle EBS deployment would subject a business to an offer like this.  “Listen Bob, you can buy 1000 licenses of XYZ for $10M or you can buy 750 licenses of XYZ for $6M, buy 400 Cloud units for $3M and we will generously throw in 250 licenses …. you’ll still have to pay support of course. You won’t get a better deal Bob, act now!”.  Yes, Oracle is willing to take a hit for the on-premises license revenue while bolstering their cloud sales by simply shuffling the Titanic deck chairs. These clients, for the most part are not interested in the Oracle cloud and will never use it other than to get a better deal during negotiations. Oracle then reports to Wall Street they are having tremendous cloud growth. Just google “oracle cloud fake bookings” to read plenty of evidence to support this.

Licensing in the Cloud

Leave it to Oracle Marketing to find a way to get even deeper into clients wallets – congratulations they’ve found a new way in the “Cloud”.  Oracle charges at least 2X more with Oracle licenses on Intel servers that run in Authorized Cloud Environments (ACE). You do not license Oracle in the cloud using the on-premises licensing factor table.  The more VM’s running in a ACE,  the more you will pay vs an on-premises deployment. To properly license an on-premises Intel server (remember, it is always an underlying proof that Oracle on POWER servers is the best solution) regardless if virtualization is used, assuming a 40 core server, would equal 20 Oracle Licenses (Intel licensing factor for Intel servers is 0.5 per core). Assume 1 VMware server, ignoring it is probably part of a larger vSphere cluster.  Once licensed, clients using VMware could theorectially run Oracle as many VM’s as desired or supported by that server. Over-provision the hell out of it – doesn’t matter. That same workload in an ACE, you pay for what amounts to every core.  Remember, if the core resides on-premises it is 1 Oracle License for every 2 Intel cores but in a ACE it is 1 OL for 1 core.

AWS
Putting your Oracle workload in the cloud?  Oracle license rules stipulate if running in AWS, it labels as vCPU’s both the physical core and the hyperthread. Thus, 2 vCPU = 1 Oracle License (OL). Using the same 40 core Intel server mentioned above, with hyperthreading it would be 80 threads or 80 vCPU.  Using Oracle’s new Cloud licensing guidelines, that would be 40 OL.  If this same server was on-premises, those 40 physical cores (regardless of threads) would be 20 OL ….. do you see it?  The licensing is double!!!   If your AWS vCPU consumption is less vs the on-premises consumption you may be ok. As soon as your consumption goes above that point – well, break out your checkbook.  Let your imagination run wild thinking of the scenarios where you will pay for more licenses in the cloud vs on-prem.

Azure
Since Azure does not use hyperthreading, 1 vCPU = 1 core.  The licensing method for ACE’s for Azure or any other ACE if hyperthreading is not used, 1 vCPU = 1 OL.  If a workload requires 4 vCPU, it requires 4 OL vs the 2 OL if it was on-premises.

Three excellent references to review. The first is Oracle’s Cloud licensing document. The second link is an article by Silicon Angle giving their take of this change and the last link is for a blog by Tim Hall, a DBA and Oracle ACE Director sharing his concerns. Just search for this topic starting from January 2017 and read until you fall asleep.

Oracle
Oracle offers their own cloud and as you might imagine, they do everything they can to favor their own cloud thru licensing, contract negotiations and other means.   From SaaS, IaaS and PaaS their marketing machine says they are second to none whether the competition is SalesForce, Workday, AWS, Azure or any other.  Of course, analysts, media, the internet nor Oracle earnings reports show they are having any meaningful success – to the degree they claim.

Most recently, Oracle gained attention for updating how clients can license Oracle products in ACE’s as mentioned above.  As you might imagine, Oracle licenses its products slightly differently than in competitors clouds but they still penalize Intel and even SPARC clients, who they’ll try to migrate into the cloud running Intel (since it appears Oracle is abandoning SPARC).  The Oracle Cloud offers clients access to its products on a hourly or monthly in a metered and non-metered format on up to 4 different levels of software. Focusing on Oracle DB, the general tiers are Standard, Enterprise, High-Performance and Extreme-Performance Packages. Think of it like Oracle Standard Edition, Enterprise Edition, EE+tools, EE+RAC+tools.  Oracle also defines the hardware tier as “Compute Shapes“. The three tiers are General Purpose, High-Memory or Dedicated compute

Comparing the cost of an on-premises perpetual license for Oracle Enterprise  vs a non-metered monthly license for the Enterprise Tier means they both use Oracle Enterprise Edition Database. Remember a perpetual license is a one-time purchase, $47,500 for EE DB list price plus 22% per year annual maintenance.  The Enterprise tier using a High-memory compute shape in the Oracle cloud is $2325 per month.  This compute shape consists of 1 OCPU (Oracle CPU) or 2 vCPU (2 threads / 1 core).  Yes, just like AWS and Azure, Intel licensing is at best 1.0 vs 0.5 for on-premises licensing per core. Depending how a server might be over-provisioned as well as the fact an on-premises server would be fully licensed with 1/2 of its installed cores there are a couple of ways clients will vastly overpay for Oracle products in any cloud.

The break-even point for a perpetual license + support vs a non-metered Enterprise using High-memory compute shape is 30 months.

• Perpetual license
  • 1 x Oracle EE DB license = $47,500
  • 22% annual maintenance = $10,450
  • 3 year cost: $78,850
• Oracle Cloud – non-metered Enterprise using High-Memory shape
  • 1 x OCPU for Enterprise Package for High-Compute = $2325/mo
  • 1 year cloud cost = $27,900
  • 36 month cost: $83,700
• Cross-over point is at 30 months
  • $79,050 is the 30 month cost in the Cloud
• An Oracle Cloud license becomes significantly more expensive after this.
  • year 4 for a perpetual license would be $10,470
  • 12 months in year 4 for the Cloud license would be $27,900
  • Annual cost increase for a single cloud license over the perpetual license = $17,430
• Please make your checks payable to “Larry Ellison”

Oracle revenue’s continue to decline as clients move to purpose-built NoSQL solutions such as MongoDB, RedisLabs, Neo4j, OrientDB, Couchbase as well as SQL based solutions from MariaDB, PostgreSQL (I like EnterpriseDB) even DB2 is a far better value.  Oracle’s idea isn’t to re-tool by innovating, listening to clients to move with the market. No, they get out their big stick – follow the classic mistake so many great clients have done before them which is not evolve while pushing clients until something breaks.   Yes, Boot Hill is full of dead technology companies who failed to innovate and adapt. This is why Oracle is in complete chaos.  Clients beware – you are on their radar!

 

 

HPE; there you go again! Part 1

Updated Sept 05, 2016: Split the blog into 2 parts (Part 2). Fixed several typo’s and sentence structure problems. Updated the description of the Superdome X blades to indicate they are 2 socket blades while using Intel E7 chips.

It must be the season as I find myself focused a bit on HPE.  Maybe it’s because they seem to be looking for their identity as they now consider selling their software business.  This time though, it is self-inflicted as there has been a series of conflicting marketing actions. From what they say in their recent HPE RAS whitepaper about the poor Intel server memory reliability stating in the introductory section that memory is far and away the highest source of component failures in a system.  Shortly after that RAS paper is released, they post a blog written by the HPE Server Memory Product Manager stating “Memory Errors aren’t the end of the World”.  Tell that to the SAP HANA and Oracle Database customers, the latter which I will be discussing in this blog.

HPE dares to step into the lion’s den on a topic with which it has little standing to imply it is an authority how Oracle Enterprise software products are licensing in IBM Power servers.  As a matter of fact, thanks to the President of VCE, Chad Sakac for acknowledging that VMware has a Oracle problem.  On August 17^th, Chad penned what amounts to an open letter to Larry & Oracle begging them …. No, demanding that Larry leave his people alone.  And, by “his people”, I mean customers who run Oracle Enterprise Software Products licensed by the core on Intel servers using VMware.

Enter HPE with a recent blog by Jeff Kyle, Director of Mission Critical Solutions.  He doesn’t distinguish if he is in a product development, marketing or sales role.  I would bet he it is the latter two as I do not think a product developer would put themselves out like Jeff just did.  What he did is what all Intel marketing teams and sellers have done from the beginning of compute time when the first customer thought of running Oracle on a server that wasn’t “Big Iron”.

Jeff sets up a straw man stating “software licensing and support being one of the top cost items in any data center” followed by the obligatory claim that moving it to an “advanced” yet “industry-standard x86 servers” will deliver the ROI to achieve the goals of every customer while coming damn close to solving world hunger.

Next is where he enters the world of FUD while also stepping into the land of make-believe.  Yes, Jeff is talking about IBM Power technology as if it is treated by Oracle for licensing purposes the same as an Intel server, which it is not.  You will have to judge if he did this on purpose or simply out of ignorance.  He does throw the UNIX platforms a bone by saying they have “excellent stability and performance” but stops there as only to claim they cost more than their Industry standard x86 server counterparts.

He goes on to state UNIX servers <Hold Please> Attention: For purposes of this discussion, let’s go with the definition that future UNIX references = AIX and RISC references = IBM POWER unless otherwise stated.  As I was saying, Jeff next claims AIX & POWER are not well positioned for forward-looking Cloud deployments continuing his diminutive descriptors suggesting proper clients wouldn’t want to work with “proprietary RISC chips like IBM Power”. But, the granddaddy of all of his statements and the one that is complete disingenuous is:  <low monotone voice> “The Oracle license charge per CPU core for IBM Power is twice (2X) the amount charged for Intel x86 servers” </low monotone voice>.

In his next paragraph, he uses some sleight of hand by altering the presentation of the traditional full List Price cost for Oracle RAC that is associated with Oracle Enterprise Edition Database.  Oracle EE DB is $47,500 per license + 22% maintenance per year, starting with year 1.  Oracle RAC for Oracle EE EB is $23,000 per license + 22% maintenance per year, starting with year 1.  If you have Oracle RAC then you would by definition also have a corresponding Oracle EE DB Licenses.  The author uses a price of $11,500 per x86 CPU core and although by doing he isn’t wrong per se, I just do not like that he does not disclose the full license cost of #23,000 up front as it looks like he is trying to minimize the cost of Oracle on x86.

A quick licensing review. Oracle has an Oracle License Factor Table for different platforms to determine how to license its products that are licensed by core. Most modern Intel servers are 0.5 per License.  IBM Power is 1.0 per License.  HP Itanium 95XX chip based servers, so you know also has a license factor of 1.0.  Oracle, since they own the table and the software in question can manipulate it to favor their own platforms as they do, especially with the SPARC servers.  It ranges from 0.25 to 0.75 while Oracle’s Intel servers are consistent with the other Intel servers at 0.5.  Let’s exclude the Oracle Intel servers for purposes of what I am talking about here for reason I said, which is they manipulate the situation to favor themselves. All other Intel servers “MUST” license ALL cores in the server with very, very limited exceptions “times” the licensing factor which is 0.5.  Thus, a 2 x 18 core socket would have 36 cores. Ex: 2s x 18c = 36c x 0.5 License Factor = 18 Licenses.  That would equal 18 Oracle Licenses for whatever the product being used.

What Jeff does next was a bit surprising to me.  He suggests customers not bother with 1 & 2 socket Intel “Scale-out” servers which generally rely on Intel E5 aka EP chipsets.  By the way, Oracle with their Exadata & Oracle Database Appliances now ONLY use 2 socket servers with the E5 processors; let that sink in as to why.  The EP chips tend to have features that on paper have less performance such as less memory bandwidth & fewer cores while other features such as clock frequency are higher, a feature that is good for Oracle DB.   These chips also have lower RAS capabilities, such as missing the MCA (Machine Check Architecture) feature only found in the E7 chips.  He instead suggests clients look at “scale-up” servers which commonly classified as 4 sockets and larger systems.  This is where I need to clarify a few things.  The HP Superdome X system, although it scales to 16 sockets, does so using 2 socket blades.  Each socket uses the Intel E7 processor, which given this is a 2 socket blade is counter to what I described at the beginning of this paragraph where 1 & 2 socket servers used E5 processors.  The design of the HP SD-X is meant to scale from 1 blade to 8 blades or 2 to 16 sockets which requires the E7 processor.

With the latest Intel Broadwell EX or E7 chipsets, the number of cores available for the HD SD-X range from 4 to 24 cores per socket.  Configuring a blades with the 24 core E7_v4 (v4 indicates Broadwell) equals 48 cores or 24 Oracle Licenses.  Reference the discussion two paragraphs above.  His assertion is by moving to a larger server you get a larger memory capacity for those “in-memory compute models” and it is this combination that will dramatically improve your database performance while lowering your overall Total Cost of Ownership (TCO).

He uses a customer success story for Pella (windows) who avoided $200,000 in Oracle licensing fees after moving off a UNIX (not AIX in this case) platform to 2 x HPE Superdome X servers running Linux.  This HPE customer case study says the UNIX platform which Pella moved off 9 years ago was actually a HP Superdome with Intel Itanium processors server running HP-UX.  Did you get this? HP migrated off their own 9-year-old server while implying it might be from a competitor – maybe even AIX on Power since it was referenced earlier in the story.  That circa 2006 era Itanium may have used a Montecito class processor. All of the early models before Tukwila were pigs, in my estimation.  A lot of bluff and hyperbole but rarely delivering on the claims.  That era of SD would have also used an Oracle license factor of 0.5 as Oracle didn’t change it until 2010 and only on the newer 95xx series chips.  Older systems were grandfathered and as I recall as long as they didn’t add new licenses they would remain under the 0.5 license model.  I would expect a 2014/2015 era Intel processor would outperform a 2006 era chip, although if it would have been against a POWER5 1.9 or 2.2 GHz chip I might call it 50-50 J .

We have to spend some time discussing HP server technology as Jeff is doing some major league sleight of hand as the Superdome X server supports a special hardware partitioning capability (more details below) that DOES allow for reduced licensing that IS NOT available on non-Superdome x86 servers or from most other Intel vendors unless they also have an 8 socket or larger system like SGI – oh wait, HP just bought them.  Huh, wonder why they did this if the HPE Superdome X is so good.

Jeff then mentions an IDC research study; big deal, here is a note from my Pastor that says the HPE Superdome is not very good; who are you going to believe?

Moving the rest of the blog to Part 2.

 

 

Intel Vendors & VMware have a Oracle Problem

Houston, we have a problem!

Intel server vendors Dell, VCE, HPE, Cisco, Lenovo, Fujitsu, Hitachi, Huawei, white box vendor Super Micro and any other server vendor using Intel chipsets have a problem if their customers use VMware to host Oracle Enterprise products (ie Database).

What’s “their” problem

In a nutshell, Oracle’s position is that customers running their Enterprise products like Oracle Enterprise Edition Database,  licensed by core (all cores in the server x 0.5) in a VMware environment must license every core on every server in which that Oracle workload could ever potentially reside managed by vCenter.  Server vendors, VMware, consultants and so on have a vested interest for Oracle to not do this because this Oracle tax is an extreme approach with their licensing terms that concern customers if they are running Oracle on Intel servers for fear Oracle will initiate a LMS audit leading to a substantial license settlement.

Quick Background

In my previous blog I wrote about “Intel; the Great Charade” where I discuss  each new generation of Intel processors having less performance per core than the previous generation.  As you read this and the ones referenced in this article (VCE & HoB) keep this ‘per core’ licensing approach in mind as this topic is central to how Oracle (typically) licenses its enterprise products.  For example, if a clients current server models are Sandy Bridge or Ivy Bridge era servers and plan to upgrade to the latest generation Broadwell you actually decrease the per core performance while increasing the number of cores per socket (if staying with the same SKU). Meaning 12 Ivy Bridge cores requires a little over 13 rounded up to 14 Broadwell cores to deliver equal performance. You don’t upgrade to get equal performance so you now have to move to a 16 or maybe 18 core SKU to gain additional socket performance or go with a higher frequency & lower core per socket SKU to obtain receive more performance per core….but now do you have enough overall performance?  To summarize my previous blog: It takes more cores from Haswell or Broadwell to equal the performance of the previous generation chips.  Since this increase in performance is at the socket and not with the core or thread (where most databases almost almost always prefer a stronger core vs more cores let alone weaker cores in a socket).  Since Oracle’s license calculation on Intel is to license all of cores in the server x 0.5 you may end up buying 1 or 2 extra Oracle licenses for every upgraded Intel server running VMware so be sure to factor that into your budget.

Who is complaining?

I could write the rest of this blog on this topic alone; around the right and wrong of Oracle’s licensing methods in VMware environments but I’ll defer to the thousands already available on this very topic.  This is not the reason I am writing this blog but to call out the self-serving and irresponsible Call-to-Action from House of Bricks and leader of a major CI player; VCE and to discuss why Oracle has no incentive to stop doing what they are doing.

Chad Sakac, the President of VCE which is the the Converged Infrastructure (CI) arm of EMC and soon Dell as the acquisition of EMC should be complete any day.  He is a regular blogger and in my opinion a master of marketing, technology & motivation.  On August 17, 2016 Chad wrote a blog titled “Oracle, I’m sad about you, disappointed in you, and frustrated with you.” in which he lays out how Intel server customers running Oracle Enterprise products, most often Oracle Enterprise Edition Database, are fed-up with Oracle’s abusive licensing tactics when Oracle Enterprise Edition products are installed and running in VMware.  He passionately pleads years of Oracle licensing frustration on behalf of clients while challenging clients to stand up to Oracle and not let them be bullied anymore.  He admits to selfishly partnering with House of Bricks (HoB), a VCE partner by funding their analysis on this situation.  HoB has been a leading voice in this fight in there own right so receiving compensation from VCE check was the proverbial icing on the cake IMHO as they were fighting the fight anyway.  What is VCE’s angle? They either have, or are losing  business due to clients fear of running Oracle workloads using VMware vSphere & vCenter.  There must be enough business at stake or EMC / VCE is desperate enough (not being critical here, just observing) to force them down this path to take such a in your face approach to Oracle.

House of Bricks, who is VCE’s partner and author of the whitepaper had a generally fair and moderated tone throughout the whitepaper.  That said, I do find they are irresponsible by encouraging VMware customers who are running  Oracle Enterprise Edition products licensed by core/processor (not socket or NUP) to run in configurations which are in direct conflict with Oracle’s standard licensing practices.  I’m not arguing the merits, fairness or legality of those licensing practices so save your comments.

Fight Mr Customer So We Can Sell You More!

Simply stated, Chad Sakac, the President of VCE and House of Bricks are actively encouraging system administrators, DBA’s and IT organizations to not only defend your use of Oracle Enterprise products in VMware environments, VMware clusters and VMware environments managed under vCenter but also to license Intel servers using sub-capacity licensing, using the BIOS to limit access to sockets or cores, only license the cores being used by Oracle.  Do these things and stand up to Oracle.  Do it for you….do it for us….just do it!  Of course, VCE funded the HoB paper but they won’t be funding your legal case (or bills) with Oracle.  All of this “encouragement” while at the same time promoting EMC / VMware / VCE products in lieu of traditional Oracle availability & replication products seems a little disingenuous…maybe….why not just keep your argument on the complaint of Oracle licensing with VMware?.  But instead, among many “do this instead of that” statements such as liminating Oracle RAC and use VMware HA and consider EMC RecoverPoint / SRDF in lieu of Oracle Active Data Guard (ADG).

Multiple agenda’s

Much of the HoB whitepaper feels like a marketing slick for EMC / VCE products. Then to have Chad be the front man out front crying on behalf of all customers seems a little too self-serving.

My Good Buddy Larry

Now back to Oracle….everybody knows I am NO Oracle fan.  A good day is any day I beat Oracle (anybody beats Oracle) or reduces their revenue.  But, with regard to Oracle’s practices on how they license their Enterprise products in a VMware environment, they have ZERO (0) motivation to loosen their licensing rules given Intel’s continued growth in the marketplace – Oracle is in the drivers seat!  Oracle wants customers to buy infrastructure from them running OracleVM with Oracle Linux hosting the Oracle software stack.  Oracle receives ALL of the Sales & Support dollars this way.  In addition to this, Oracle is predisposed to litigate.  Larry likes to fight!  HP and now HPE, SAP, Google (2 suits, going to a 3rd), Rimini Street, Oregon Healthcare, Mars and many more.  The Oracle v Mars case is a recent example of how Oracle goes after customers using their License Management Service (LMS) group to drive license revenue thru audits.  “Mars stated that Oracle was unwilling to “come to a mutually agreeable process” for completing an audit. Oracle then sent Mars a letter stating Mars had materially breached its license agreement”.  The greatest leverage clients have is to move off of Oracle products (hardware & software) to alternative solutions; specifically database variants such as IBM DB2, Microsoft SQL Server or Open Source alternative PostgreSQL from EnterpriseDB not to mention the many NoSQL alternatives that probably do a far better job.

Alternatives

If VCE really wanted to partner with an enterprise quality commercial-grade database technology to help clients run VMware with sub-capacity licensing for just the servers where the workloads are running and find an alternative to Oracle, they should look at IBM’s DB2 . DB2 is available in multiple editions from a free edition to Advanced Enterprise Server Edition.  What makes it different and better than both Oracle or SQL Server is that AESE, for example, includes many of the products & features that a client desires of Oracle Enterprise Edition products yet have to pay for À la carte.  DB2’s AESE cost $56,210 (list price for 70 PVU) per license which would match up against the Oracle Enterprise Edition portfolio which when you add up those products cost over $225K (Note: DB2 ESE is a level down from AESE, cost less and probably meets 90% of the customers requirements so the story just gets better).  DB2 always includes its first year of maintenance then 20% each year thereafter while Oracle always charges 22% for the first year then 22% each year thereafter.  Of course, DB2 runs 2X faster with Linux on POWER vs Intel. Clients can try it out for free in SoftLayer for 30 days running Linux on a OpenPOWER server.  Since LoP isn’t the topic of this blog, I’ll save that for another day but know that at least both Intel with VMware and IBM POWER servers support sub-capacity licensing with virtualization.

I didn’t write this blog to be a shill for IBM’s DB2 either, it just came to me as I was reading the HoB paper as it felt like they were trying to slyly present SQL Server as a more agreeable alternative to Oracle – maybe they are … either way thought I would mention DB2 for some balance.

There is ONE Platform …

At the end of the day, clients have a choice if they run Oracle products such as PeopleSoft, JD Edwards, Oracle Apps, Oracle E-Business Suite (EBS) or standalone Oracle Enterprise products like Database, RAC, WebLogic and many others.  Clients can run Oracle on Intel with VMware then surely deal with the risk and issues discussed by Chad and House of Bricks OR clients could run Oracle on the only platform which controls Oracle licensing without all of the consternation, debate and angst; IBM POWER servers running AIX.  For those who have read this far and were begging to say “But POWER servers have a core license factor 2X of  Intel so they cost twice as much”.  Enough please! I may hire House of Bricks to write a paper to put an end to this FUD, myth and farce. With POWER8 outperforming Intel servers generally around 2X per core it eliminates this argument right here. But, since we are talking about licensing a product at the core level it is important to remember that POWER servers support sub-capacity licensing natively, without debate from Oracle.  Last and most importantly, IBM’s Power Hypervisor suite, called PowerVM manages the compute resources more efficiently where it scales the 2X performance per core advantage typically increasing it up to 4X, 8X, 12X, even 20X (your mileage will vary).  This isn’t a performance advantage as much as it is an efficiency statement.  I call it the “Total Cost of Efficiency” as it takes into account the TCA, Performance advantage & Hypervisor efficiency and depending on the discussion, years 2-5 maintenance which is TCO.  I have personally sized, architected and delivered these solutions to customers who have in turn realized these very savings.

Now the Call-to-Action!

If you believe VMware & Intel are a critical part of your business identity that make your products better then continue using them with your Oracle products. You will pay more (compared to POWER) due to lower performance & less efficiency and pay the Oracle tax.  If you view IT as an enabler to your bottom line and use the right tool for the job then give me a call as I can help you as I have helped dozens of others save $100K’s to $M’s with IBM server technology.   Oh, and for those last few sharpshooters who want to remark that IBM servers are more expensive go ahead and save your comment.  First, I’ll shut you down by comparing a proper IBM server with the class of Intel server that you present me.  Next, we won’t go the 1 for 1 server route. As I recently showed a customer a reduction of 24 x Dell servers with 596 cores or 298 Oracle licenses to 7 x POWER8 servers with 168 cores and Oracle Licenses. My 7 servers are far less expensive than your 24 servers  not to mention the infrastructure required to support it (power cables, LAN/SAN cables, switch ports, cooling, etc). What makes me different is I show you how it’s possible to save  significant money running Oracle on IBM servers. What makes Ciber different is we have an Oracle consulting practice to help you implement, migrate or optimize your environment.

Will a Dell acquisition of EMC be another HP-Compaq disaster?

There is a feeding frenzy on the M&A speculation regarding Dell buying EMC.  How much will it cost, who is impacted and how big of an impact it will be?  Although the speculation is interesting, the 800 lb gorilla in the room is “What’s going on at EMC” to warrant this.

Working  in IT sales, I work and compete with EMC and their partner organizations; primarily VCE and VMware and their recently divorced partner, Cisco.  Following EMC and VMware at their respective vendor conferences; EMC World and VMworld they have an extremely large and dedicated following.  EMC sellers are some of the most aggressive and by extension successful compared to vendors.  I personally follow many of their technologist on social media reading their blogs and tweets – I view them as a very formidable competitor and occasional partner (even though I am not EMC’s biggest fan).

Why does a company whose products and partnerships are either #1 or leading in their respective areas desperate to find a company to acquire or possibly merge with them? It was shocking to hear HP’s name mentioned over a year ago.  They are an absolute train wreck as we now see.  If I was an EMC company officer or board member, I would not want to see my company consumed by the likes of HP.  Cisco makes sense but I don’t know how deep the recent divorce and hurt feeling goes.  Lenovo doesn’t make sense, nor Hitachi.  Fujitsu….well, they have their Oracle relationship and they seem content to with who and what they are? What about Oracle….EMC as a company has its own ego, I don’t think Larry’s ego and EMC’s ego would mesh. Although, Oracle’s storage is abysmal and this would elevate them to the leaders quadrant immediately. Maybe that would give Oracle the needed reason to finally kill off SPARC given its unending failures. How about IBM? Frankly, they have a great storage portfolio, they simply do not have comparable storage sellers to EMC. EMC’s software would fall into IBM’s vast portfolio then either rise to the top or fade into oblivion. I could see this making sense for IBM but don’t know if IBM has the stomach to ingest a company with such a different culture.  From Dell’s perspective, an EMC acquisition makes sense.  I see them having their most success in the SMB and academic markets then probably state and local governments.  At least in the US with corporate America, Dell is not a respected enterprise brand.  Buying EMC would give them immediate credibility, access to many customers data centers they are not in today.  They would have to decide what to do with their Compellent and EqualLogic products. I would probably keep the EqualLogic product family as it seems to service their low end SMB space, a space that I don’t think EMC cares much about.  Sell Compellent to get cash for the investments made as well as to strengthen the books.

From EMC’s perspective, they would be acquired by a company known for inexpensive laptops and decent desktops (How’s that market doing these days?) and reselling Lexmark printers (who makes them now?).  Only in the last couple of years do I occasionally hear Compellent’s name being mentioned and it is usually to smaller shops with lots of VMware….let’s say shops that aren’t very sophisticated. Their is still a relationship with Cisco both with EMC to Cisco but also with vBlocks.  VMware has their EVO:Rail and Dell has their partnership with Nutanix.  Lastly, their is VMware.  Here’s the irony of what is going on: Some people want EMC to spin off VMware while others want EMC to keep them to strengthen and shore up the Federation.  If the stories are right, Dell would buy EMC then sell off VMware to reclaim much of its acquisition cost.

We will  soon see if there is any truth to the rumors about Dell acquiring EMC. if they are true, will this be a repeat of what happened to HP after merging with Compaq (HPQ after all)?  They failed to integrate the many technologies and companies over two decades.  Some cost a fortune and often the cultures were vastly different.  HP is different from both Dell and EMC in that they made one bad M&A decision after another.  DEC, Tandem, Compaq, Autonomy, 3PAR, 3COM and Palm (just from memory).

Maybe a Dell acquisition of EMC makes sense but it will take a couple of years to see what products survive and what doesn’t. This of course will be disconcerting to customers who have invested in their respective technologies. IBM was often the source of FUD attacks claiming the end of every one of their businesses as well.  It is my contention that is how they have made it to 100+ years.  I hope we will hear someday what was really going on at EMC to warrant their ongoing search for a suitor.  The last mega merger of a commodity company to a enterprise company didn’t turn out so well: HPQ!