Performance, Scalability and Architecture

Andreas Grabner

Subscribe to Andreas Grabner: eMailAlertsEmail Alerts
Get Andreas Grabner: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: Application Performance Management (APM), DevOps Journal

Blog Post

Detecting Bad Deployments [#APM with #DevOps] | @DevOpsSummit

Resource usage monitoring is as important as response time and throughput.

Detecting Bad Deployments on Resource Impact & Not Response Time

This story came in from Joseph - one of our fellow dynaTrace users and a performance engineer at a large fleet management service company. Their fleet management software runs on .NET, is developed in-house, is load tested with JMeter and monitored in Production with dynaTrace. A usage and configuration change of their dependency injection library turned out to dramatically impact CPU and memory usage while not yet impacting end user experience. Lessons learned: resource usage monitoring is as important as response time and throughput.

On Wednesday, July 3, Joseph's ops team deployed the latest version into their production environment. Load (=throughput) and response time are two key application health measures the application owner team has on their production dashboards. Especially after a new deployment, an unchanged behavior is what you are looking/ hoping for:

APM Lesson #1: Don't judge a deployment based on response time and throughput alone

Joseph is an experienced performance engineer, so he knew to check more than just these two metrics. Looking at host and process monitoring data he identified a 6x increase in CPU and 40% increase in process memory usage, which alarmed the folks responsible for the infrastructure that hosted the ASP.NET Services:

APM Lesson #2: Expand your alerting to resource usage and infrastructure health

Despite the fact that the app still shows the same response time and throughput this is very alarming. Why? Because it means that the application is much more expensive to run as compared to the previous build.

Whether it runs on physical, virtualized or in a cloud environment: more resource consumption means higher operational costs. And who knows what happens during peak load: will CPU and memory be enough or will it eventually impact response time and throughput? That's the reason why analyzing changes in resource consumption is as critical to performance engineering as optimizing response time and throughput.

If you are a performance engineer, application owner or developer I suggest you read the rest of the blog to learn about Joseph's best practices on analyzing and dealing with situations like this and what they changed in their development and deployment process:

Step 1: Analyzing the Impact of Memory Usage on CPU
Runtime engines such as the JVM (more on Java Enterprise Performance) or the .NET CLR (more on.NET Performance Analysis) provide detailed performance counters to start analyzing the resource usage hotspots on CPU and memory. A good start is to look at memory usage in the different heap generations as well as the activity of the garbage collector. The following screenshot shows the .NET garbage collection generations (# of Collections and Heap Size) as well as the time spent in garbage collection metrics of their production environment:

APM Lesson #3: Look at memory metrics to figure out how memory usage and GC behavior has changed

What can you read from these two graphs?

  1. Most of the objects live in generation 0 and never get promoted to an older generation. That means we are talking about short-lived objects that are probably created and cleared on a per request basis. If we saw a shift to older generations it is more likely that these objects are part of a cache.
  2. We don't seem to have a memory leak as the garbage collector seems to clear objects from Gen 0 and not promote them to higher generations, which could eventually lead to an out-of-memory crash.
  3. The increase in # of collections as well as time spent in garbage collection indicates that the application either uses more short-lived objects or these objects grew significantly in size.

Step 3: More Objects or Larger Objects or Both?
The next step is to find out what has changed in terms of objects on the heap. Joseph therefore takes regular dynaTrace memory snapshots (YouTube Tutorial) and keeps them to later compare heap usage patterns between releases. The next screenshot shows the difference between the previous version and the current version and highlights that there are not only about twice as many ASP.NET MVC controller objects on the heap but these objects are also about 6x larger than before:

APM Lesson #4: Continuously take memory snapshots to identify regressions between builds

It turned out it that this pattern was not exclusive to the Planning Controller shown in the screenshot but happened across all ASP.NET MVC Controller objects. Following the Deep Memory Leak Snapshots best practices, Joseph found the actual referenced objects that caused these controller objects to grow in size. It turned out to be related to objects that were instantiated and referenced using dependency injection.

Finding and identifying the problem was the first step which raised two more questions:

  1. Why wasn't this found earlier?
  2. Which code change caused this problem?

For answers, click here for the full article.

DevOps: Disruptive but Essential in a Cloud Computing Universe

The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.

The DevOps Summit at Cloud Expo--to be held November 4-6 at the Santa Clara Convention Center in the heart of Silicon Valley--will expand the DevOps community, enable a wide sharing of knowledge, and educate delegates and technology providers alike. Recent research has shown that DevOps dramatically reduces development time, the amount of enterprise IT professionals put out fires, and support time generally. Time spent on infrastructure development is significantly increased, and DevOps practitioners report more software releases and higher quality.

DevOps Summit content doubled from a single track in New York to two simultaneous tracks: "Dev" Developer Focus and "Ops" Operations Focus.

Join DevOps Summit Chair Andi Mann and Host Alan Shimel, November 4-6 for three days of intense DevOps discussion and focus.

We'll see you in Santa Clara! See you in November!

DevOps Summit Conference Schedule Announced! ▸ Here

Register For DevOps Summit "FREE" (limited time) ▸ Here

Website: DevOps Summit

Twitter: @DevOpsSummit

Bookmark "DevOps Journal" ▸ here

@DevOpsSummit Silicon Valley (November 4-6, 2014, Santa Clara, CA)
@DevOPsSummit New York (June 9-11, 2015, New York, NY)

DevOps Summit New York 2015 Call for Papers Now Open

Call for Papers will remain open through December 31, 2014. DevOps Summit New York 2015 will take place June 9-11, at the Javits Center, in New York City. Tracks and topics for DevOps Summit at Cloud Expo are listed below. Submit your speaking proposals here DevOps Summit Call for Papers

  • Application delivery
  • ROI and business value
  • Site reliability
  • Continuous delivery
  • Kanban, Scrum, and Agile
  • Continuous integration
  • Release scalability
  • Continuous release
  • Service virtualization
  • Operational monitoring
  • Capacity management
  • Load testing
  • Quality assurance
  • Manager and exec support
  • Release automation
  • Cultural change
  • Breaking down IT silos
  • Cloud development
  • Cloud platforms
  • Test automation
  • Teaming
  • Application security
  • API management
  • Identity and access
  • Audit and control
  • DevOps and ITIL, ISO, Six Sigma, COBIT
  • Mobile DevOps
  • DevOps for legacy systems
  • Software-defined servers
  • Network automation
  • Server automation
  • Configuration automation
  • Continuous support
  • DevOps anti-patterns
  • Enterprise DevOps
  • DevOps system architecture
  • IT orchestration
  • Containerization
  • DevOps skills and training
  • WebOps, CloudOps, ChatOps, NoOps
  • Change management

Video Highlights From DevOps Summit at Cloud Expo New York

While there are many ways to define DevOps, the goal of the concept is to be able to deliver IT solutions faster, leveraging several technology tools to add value for business. Cloud companies have demonstrated how they can manage massive pools of IT infrastructure, giving the developer a very flexible resource pool that they can leverage and IT is eager to do the same.


Is DevOps Really Changing How IT Is Working?

In this Power Panel at 2nd DevOps Summit moderated by Cloud Expo Tech Chair Larry Carvalho, Bernard Golden, VP, Strategy at ActiveState; Andi Mann, VP of Strategic Solutions at CA Technologies; John Willis, VP of Customer Enablement at Stateless Networks; Randy Bias, Co-Founder and CTO of Cloudscaling; and Vasu Sankhavaram, Chief Strategist of DevOps and Cross Portfolio Architecture Alignment within HP Software, discuss this movement that is delivering frictionless IT to the enterprise.


DevOps and Security

From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises - and delivering real results. In this DevOps Summit Power Panel (http://DevOpsSummit.SYS-CON.com/) on June 9 at our New York City studio, moderated by Andi Mann, DevOps Summit conference chair and VP of Strategic Solutions at CA Technologies, John Willis, VP of Customer Enablement for Stateless Networks; JP Morgenthal, Director, Cloud Computing Practice at Perficient, Inc.; Vanessa Alvarez, Cloud Expo Conference Chair and Marketing Team at Amazon; and Gordon Haff, Cloud Evangelist at Red Hat; discussed how security needs to be embraced and brought into the DevOps movement.


Is DevOps Changing Enterprise IT?

Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. In this DevOps Summit Power Panel on June 9 at our New York City studio, moderated by Andi Mann, DevOps Summit conference chair and VP of Strategic Solutions at CA Technologies, JP Morgenthal, Director of Cloud Computing Practice at Perficient, Inc., Gordon Haff, Cloud Evangelist at Red Hat; Larry Carvalho, Cloud Expo Tech Chair and Research Manager & Lead Analyst, Platform as a Service at IDC; and John Willis, VP of Customer Enablement for Stateless Networks; discussed how DevOps is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.

Schedule for 15th @CloudExpo, @BigDataExpo and @ThingsExpo

Are you ready to put your data in the cloud?

What is the future of security in the cloud?

Does Docker quickly advance the development of an IoT application?

What are the implications of Moore's Law on Hadoop deployments?

Get all these questions and hundreds more like them answered at the 15th Cloud Expo, November 4-6, 2014, at the Santa Clara Convention Center, in Santa Clara, CA. The Cloud ExpoBig Data Expo@ThingsExpoDevOps Summitprograms are now available for you to inspect and investigate in advance.

Our upcoming November 4-6 event in Santa Clara, California will present a total of 10 simultaneous tracks by an all-star faculty, over three days, plus a two-day "Cloud Computing Bootcamp" presented by Janakiram MSV, an Analyst with the Gigaom Research analyst network, where he covers the Cloud Services landscape.

Cloud and Big Data topics and tracks include: Enterprise Cloud AdoptionAPM & Cloud Computing | Hot TopicsCloud APIs & BusinessCloud Security | MobilityBig Data | Analytics.

@ThingsExpo content tripled from a single track in New York to three simultaneous tracks: Consumer IoTEnterprise IoTIoT Developer | WebRTC Convergence.

Schedule for Cloud Expo / Big Data Expo / @ThingsExpo ▸ Here

Now that we have published the full conference schedule, please check back for daily updates as we finalize new session abstracts by working with our distinguished faculty members. For your questions please contact us at events (at) sys-con.com. Last but not least we will announce our keynotes on the hottest subjects to be delivered by world-class speakers!

The largest 'Internet of Things' event in the world has announced "sponsorship opportunities" and "call for papers."

The 1st International Internet of @ThingsExpo was launched this June at the Javits Center in New York City with over 6,000 delegates in attendance. The 2nd International Internet of @ThingsExpo will take place November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, California, with an estimated 7,000 plus delegates attending over three days.

@ThingsExpo is co-located with 15th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading IoT industry players in the world. In 2014, more than 200 companies will be present on the @ThingsExpo show floor, including global players and the hottest new technology pioneers.

Sponsorship and Exhibit Opportunities for @DevOpsSummit Silicon Valley and New York Are Now Available
Sponsors and exhibitors of Internet of @DevOpsSummit will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for sponsors that have been guaranteed a 35-minute technical session
  • Online advertising in SYS-CON's i-Technology publications
  • Capitalize on our comprehensive marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
  • Unprecedented PR Coverage: Editorial coverage on IoT.sys-con.com, Tweets to our 75,000 plus followers, press releases sent on major wire services to over 500 combined analysts and press members who attended Internet of @ThingsExpo - making it the best-covered "Internet of Things" conference in the world

For more information on sponsorship, exhibit, and keynote opportunities contact Carmen Gonzalez by email at events (at) sys-con.com, or by phone 201 802-3021. Book both events for additional savings!

@DevOpsSummit Silicon Valley (November 4-6, 2014, Santa Clara, CA)
@DevOPsSummit New York (June 9-11, 2015, New York, NY)

Secure Your VIP Pass to Attend @DevOpsSummit Silicon Valley
DevOps Summit at Cloud Expo announced today a limited time free "Expo Plus" registration option. The onsite registration price of $600 will be set to 'free' for delegates who register during special offer period.

To take advantage of this opportunity, attendees can use the coupon code and secure their registration to attend all keynotes and general sessions, as well as a limited number of technical sessions each day of the show, in addition to full access to the expo floor.

The registration page is located here.


@ThingsExpo New York 2015 'Call for Papers' Now Open
The 3rd International Internet of @ThingsExpo, to be held June 9-11, 2015, at the Javits Center in New York City, New York announces that its 'Call for Papers' is now open. The event will feature a world class, all-star faculty with the hottest IoT topics covered in three distinct tracks.

Track 1 - Consumer IoT and Wearables: Smart Appliances, Wearables, Smart Cars, Smartphones 2.0, Smart Travel, Personal Fitness, Health Care, Personalized Marketing, Customized Shopping, Personal Finance, The Digital Divide, Mobile Cash & Markets, Games & the IoT, The Future of Education, Virtual Reality

Track 2 - Enterprise IoT: The Business Case for IoT, Smart Grids, Smart Cities, Smart Transportation, The Smart Home, M2M, Authentication/Security, Wiring the IoT, The Internet of Everything, Digital Transformation of Enterprise IT, Agriculture, Transportation, Manufacturing, Local & State Government, Federal Government

Track 3 -  Developer IoT: WebRTC, Eclipse Foundation, Cloud Foundry, Docker & Linux Containers, Node-Red, Open Source Hardware, Leveraging SOA, Multi-Cloud IoT, Evolving Standards, WebSockets, Security & Privacy Protocols, GPS & Proximity Services, Bluetooth/RFID/etc., XMPP, Nest Labs


@ThingsExpo billboard is viewed by more than 1.3 million motorists per week on Highway 101, in the heart of Silicon Valley

Help plant your flag in the fast-expanding business opportunity that is the Internet of Things: Submit your speaking proposal today here!

Download @ThingsExpo Newsletter Today ▸ Here

Chris Matthieu Named @ThingsExpo Tech Chair

Internet of @ThingsExpo named Chris Matthieu tech chair of Internet of @ThingsExpo 2014 Silicon Valley.

Chris Matthieu has two decades of telecom and web experience. He launched his Teleku cloud communications-as-a-service platform at eComm in 2010, which was acquired by Voxeo. Next he built an open source Node.JS PaaS called Nodester, which was acquired by AppFog. His latest startups include Twelephone. Leveraging HTML5 and WebRTC, Twelephone's BHAG (Big Hairy Audacious Goal) is to become the next generation telecom company running in the Web browser. Chris is currently co-founder and CTO of Octoblu.

Website: http://www.ThingsExpo.com

Twitter: http://www.Twitter.com/ThingsExpo

About SYS-CON Media & Events
SYS-CON Media (www.sys-con.com) has since 1994 been connecting technology companies and customers through a comprehensive content stream - featuring over forty focused subject areas, from Cloud Computing to Web Security - interwoven with market-leading full-scale conferences produced by SYS-CON Events. The company's internationally recognized brands include among othersCloud Expo® (CloudComputingExpo.com@CloudExpo), Big Data Expo (BigDataExpo.net@BigDataExpo), DevOps Summit (DevOpsSummit.sys-con.com@DevOpsSummit),  Internet of @ThingsExpo (ThingsExpo.com@ThingsExpo) and Cloud Computing Bootcamp (CloudComputingBootcamp.com).

Cloud Expo®, Big Data Expo® and @ThingsExpo® are registered trademarks of Cloud Expo, Inc., a SYS-CON Events company.

More Stories By Andreas Grabner

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.