Using the Cloud to Troubleshoot End-User Performance (Part 1)

Debugging performance issues is hard. Debugging end-user performance issues from a distributed production software stack is even harder, especially if you are the 3rd-party service provider for one of your clients that actually is in control of how your code is integrated into their site. There are lots of articles on the web regarding what the performance best practices are, but few, if any, that discuss the tactics of developing and improving them.

Primarily, the challenges stem from the fact that troubleshooting performance issues is always iterative. If your production operations can handle deploying test code to production on a rapidly iterative schedule, then you can stop reading this post — you are already perfect, and worthy of a cold brewski for your impressive skills.

As our team and software stack continued to grow in both size and complexity, we underwent a restructuring of our client (js, css and html) code a while back that allows us to only deliver the code that is actually needed by our client’s site and configuration. We did this by reorganizing our code into modules that can be loaded asynchronously by the browser using require.js. Effectively this took us from a monolithic js & css include that contained a bunch of unused code and styles, to something that averaged out to a much smaller deliverable that was consumed in smaller chunks by the browser.

This technique is a double-edged sword, and like all things, is best when done in moderation. Loading multiple modules asynchronously in the browser results in multiple http requests. Every HTTP request made by the browser results in some overhead spent doing the following:

DNS Lookup – Only performed once per unique hostname that the browser encounters.
Initial Connection – This is simply the time that the browser takes to establish a TCP socket connection with the web server.
SSL Negotiation – This step is omitted if the connection is not intended to be secure. Otherwise, it is just the SSL handshake of certificates.
Time to First Byte (TTFB) – This is the time starting from after the browser has sent the request to when the first byte of the response is received by the browser.
Content Download – This is the time spent downloading all the bytes of the response from the web server.

There are many great resources from Yahoo! and Google which discuss the details of best-practices for improving page performance. I won’t re-hash that great information, nor dispute any of the recommendations that the respective authors make. I will, on the other hand, discuss some tactics and tools that I have found beneficial in analyzing and iterating on performance-related enhancements to a distributed software stack.

Mission

A few months ago, we challenged ourselves to improve our page load performance with IE7 in a bandwidth constrained (let’s call it “DSL-ish”) environment. DSL connections vary in speed and latency with each ISP.

I won’t bore you with the details of the changes we ended up making, but I want to give you a flavor of the situation we were trying to solve before I talk about the tactics and tools we used to iterate on the best practices that are available all over the web.

The unique challenges here are that IE7 only allows 2 simultaneous connections to the same host at a time. Since our software distributes multiple modules of js, css and images that are very small, we were running into this 2-connection-per-hostname issue with a vengeance. Commonly accepted solutions to this involve image spriting, file concatenation and distributing or “sharding” requests for static resources across multiple domains. The sharding tactic made us realize the other constraining factor we were dealing with — the longer latency of HTTP requests on a DSL connection that gets exaggerated when making multiple DNS lookups to a larger set of distinct host names.

Tools

The tools that we used to measure and evaluate our changes affected the tactics we used – so I’ll discuss them first.

Charles Proxy

Charles Proxy is a tool that runs on all platforms and provides some key features that really aided us in our analysis. Primarily, it had a built-in bandwidth throttling capability which allowed us to simulate specific latency and upload/download conditions from our local machine. We used CharlesProxy for a rougher on-the-spot analysis of changes. CharlesProxy also allowed us to easily and quickly see some aggregate numbers of specific metrics we were interested in. In particular, we were looking for the total # of requests, total durations of all requests and the total response size of all requests. Since these numbers are affected by the rest of the code (not ours) on our client’s site – Charles allowed us to filter out the resources that were not ours, but still allowed us to see how our software behaved in the presence of our client’s code.

However, since we had multiple developers working on the project — each making isolated changes — we wanted a way to run a sort of “integration” test of all the changes at once in a manner that more closely aligned with how our software is delivered from our production servers. This led us to our next tool of choice – one that we’d never used until now.

WebPageTest.org

In it’s own words:

WebPagetest is a tool that was originally developed by AOL for use internally and was open-sourced in 2008 under a BSD license. The platform is under active development by several companies and community contributors on Google code. The software is also packaged up periodically and available for download if you would like to run your own instance.

In our case, WebPageTest provided two key things:

It’s Free
It is a useful 3rd party mediator between ourselves and others for spot-checking page performance

At a high level, WebPageTest.org controls a bunch of compute agents that live in various geographic locations of the US that are able to simulate bandwidth conditions according to your specifications (under the hood it uses DummyNet). It allows you to request one of it’s agents to load your page and interact with your site by simulating link clicks (if necessary) and monitors and captures the results for detailed analysis by you later. This tool is a great way for you to use an external entity to verify your changes and have a consistent pre & post benchmark of your page’s performance.

Of course, having some random machine on the web poke your site means that your changes must be publicly accessible over the web. Password protection is fine since you can use WPT to script the login, but IMHO is non-ideal as that is not part of the normal end-user experience.

Tactics

Now that we have a good handle on the tools we used – we should discuss how we put them to work. Stay tuned for part 2, where we will explore the tactics for using these tools together effectively.

Grilling up an API

BBQ is a religion in Austin. Everyone has their opinion on who serves up the best BBQ. Debates between people defending their choices have been known to last into the wee hours of the night. Friendships have been ruined, and neighbors turned into enemies (okay, I might have made that last bit up…but you get my point).

APIs are also like a religion to many in the developer community. Developers spend their precious time using the tools and APIs that companies create. The easiest tools to use will be the ones they turn to consistently – and tell their friends about. At Bazaarvoice, we are hyper-focused on how to make our API and Platform the best set of tools around.

But how do you “serve up” good API? To answer that, let’s borrow an analogy from the world of BBQ.

Imagine you wanted to make a BBQ dinner, and you came to Bazaarvoice for help. We could help you in a few different ways:

Method #1: We can provide you with the raw ingredients & materials you need – e.g. spices to make your sauce, sticks to build your fire, and of course – a cow.

Method #2: We can provide you with some pre-packaged ingredients & items – e.g. a bottle of BBQ sauce, a grill, and some prime cuts of meat.

Method #3: We can provide you with a menu from the Salt Lick, as well as the number to their delivery service.

So what does this translate to (aside from a yummy BBQ dinner)?

Method #1: High innovation, high support costs, and low adoption.

Method #2: Medium innovation, medium support costs, and medium adoption.

Method #3: Low innovation, low support costs, and high adoption.

At Bazaarvoice, we aim to provide the developer community with tools that support all three of the methods above:

Method #1: Our API gives developers fine-grained control over the information they can request, the filters they can specify, etc. However, this flexibility comes at a cost. Developers will have to understand our object model and syntax to take full advantage of the API, and we at Bazaarvoice need to provide training and documentation to help with this process.

Method #2: Our API documentation always starts off with example API calls and popular use cases. These “pre-packaged” examples can help you skip straight to the API calls that will get the job done.

Method #3: We have reference apps available to download, and we will continue to add more over time. These reference apps serve two purposes. First, you can download the apps, enter your API credentials, and be off and running (just like BBQ takeout!). Second, you can use these apps as a learning tool to help you get familiar with the Bazaarvoice API faster.

Like grilling up BBQ, it is hard to satisfy everyone. But when you get your product just right, you can turn customers into dedicated fans. So in conclusion – when you see an employee of Bazaarvoice feasting away at one of the many popular local BBQ joints, feel confident knowing that we are hard at work.

Extremenly superb way of comparing APIs with BBQ though firstly was quite confussed while reading this article.. So finally can say enjoyed both Bazaarvoice API and Austin BBQ 😉

The Tools We Use to Innovate in Bazaarvoice Labs (Part 2)

In the previous post, I provided a rundown on what Bazaarvoice Labs is, our process and why it is important to have flexibility in our toolset choices. I now want to give you some tool examples in the following categories:

Operational Tools
Server-side Application Development Environments
Data Storage and Management
Client-side Tools
Measurement Tools

Operational Tools

Amazon EC2: Well, duh. I mentioned that we need to seamlessly transition from internal prototypes to live running pilots and by using EC2, Elastic Load Balancer and creating a set of mostly standardized AMIs, we’re able to get a machine up and running to demo a prototype or scale out to supporting hundreds of thousands of requests almost instantly. Key to our use of the EC2 is the fact that it has a very robust API and tools like boto so we can automate just about everything that we do. This is important since it’s well documented that EC2 instances can go up and down without rhyme or reason. Which brings me to my next operational tool…
Cloudkick: We use Cloudkick for basic monitoring. Its UI is simple and it just plain works. Given how frequently we take services and applications up and down in EC2, it’s really nice to have an easily configurable, straightforward monitoring solution to rely on.

Server-side Application Development Environments

Ruby on Rails and Django: While we’ve experimented with microframeworks like Flask, sometimes when you’re moving fast and prototyping, you don’t know exactly what you need or when you’re going to need it. You may not want to think about what ORM or templating language to use or want to re-invent how user sessions are handled and it’s times like these that a nice full-stack web application framework comes in handy. Why both though? Well, quite simply, some engineers on our team prefer Ruby and some (most) prefer Python. This is where our one engineer, one project comes in handy. We work with the tools that will make us fastest. Ultimately, if someone needs to step up and lend a hand on a project when someone is on vacation, we’re all polyglots and can get our hands dirty in any language or framework necessary. The Facebook apps referenced above were written in Rails and the very, very high traffic pilot that we ran with TurboTax was written with Django (as was our Customer Intelligence product).
Node.js: The evented asynchronous server built on Google’s V8 Javascript engine. Node is a great tool to use when you’re building an application that needs to pull data in from multiple HTTP-based APIs and mash it together. Its performance is remarkable and it allows a developer to work in the same language in both the client and on the server. While some people think server-side JS is a fad, I think Node is leading a revolution in how people build and think about web applications. Please note that Node is so much more useful than for just building webapps. It can be used, for example as a very effective proxy as well (see Joe Stump’s answer about what technologies SimpleGeo uses on Quora). Data for Travelocity’s Social Connect Discovery pilot is served from Node.js backed with the Bazaarvoice Developer API and custom indices stored in Redis.

Data Storage and Management

ElasticSearch: We’re no strangers to Lucene-based search and data stores at Bazaarvoice. Most of our core platform’s displays are backed by queries made to SOLR. However, unlike SOLR, ElasticSearch is schema free and therefore really nice to use for prototyping and pilots where you’re not sure of the kinds of data that you’ll be wanting to index. There are some gotchas with this approach but for Labs projects, we’ll take the flexibility it offers. As a side note, it’s amazing how often Lucene-based tools are left out of the NoSQL discussion (In fact, my colleague RC Johnson did a SXSWi presentation on this). The search functionality in our Ask and Answer for Facebook pilot with Nikon is driven out of ElasticSearch.
MongoDB: We’ve used MongoDB in any number of Labs pilots at this point. Most notably, it drives the leaderboard and newsfeed functionality in our Ratings and Reviews for Facebook pilot with Benefit Cosmetics and also the majority of our new product discovery pilot application that we’re running with Sam’s Club in Facebook.

Client-side Tools

Dust: Dust is a Javascript templating library well suited to asynchronous applications. We like Dust because it’s a flexible and easy to use templating language, it integrates well with server-side JS tools like Node and allows you to pre-compile your templates for great performance.
Protovis: Protovis is an excellent visualization library. It’s declarative and very easy to build complex, interactive visualizations while still having a high degree of flexibilty over how those visualizations are rendered. We use Protovis to create what I believe are visualizations that are way beyond typical for an analytics tool in our Customer Intelligence product.

Measurement Tools

Google Analytics: It’d be tough to tell where we’d be without Google Analytics. It’s got its obvious uses, but also has comprehensive APIs that allow you to call custom events, set variables and then suck the data back out as necessary. This allows us to track specific actions that a user takes and to set up funnels based on those actions (even when the actions are clicks within a page vs. full page views).
Mixpanel: Mixpanel is a great alternative to Google Analytics. Many of our projects in Bazaarvoice Labs take the form of Javascript plugins or widgets that don’t conform to the traditional page-view-first mentality of most web analytics. Mixpanel focuses much more on tracking individual events that a user takes either in-page or across pages. Their API for doing this is very easy to use and it has the added benefit of being realtime which means you don’t have to wait a few hours to start seeing results from that code change that you just launched.

Of course, no project, prototype or pilot would get off the ground in Bazaarvoice Labs if we couldn’t get at our customer’s data. In order to maintain agility, all Bazaarvoice Labs projects are written as free-standing applications that are not part of our core application stack (a somewhat traditional J2EE application built on Spring MVC). Early on in Labs, even though we had direct access to our databases, we knew we needed to maintain separation between our core stack and Labs applications. Since we maintain a very complex set of business rules that are configurable on a per client basis around content submission and display, if we were to write directly to the databases, there’d be a high risk that we’d compromise data integrity. Generally, we’d use our existing XML API for submission (because it was obvious that trying to write data into the DBs from a separate application was a recipe for disaster) but we’d still use replicas of our core MySQL database clusters for display. That was okay but there were still some business logic mistakes made in the display of content (unacceptable when your pilot clients are some of the biggest online retailers around). In order to get around this, we created a new API that supported significantly higher degree of queryability, JSON and JSON-P data formats and had much lighter weight responses. This allows Bazaarvoice Labs to talk to our core data sets in a much more efficient manner and be assured that business rules are followed. This new API has now be productized as The Bazaarvoice Developer API. We will often create new, experimental method calls or create application local data indexes, but every single Bazaarvoice Labs project leverages this API heavily.

I hope I’ve given you a good overview of how Bazaarvoice Labs operates and the tools that keep us humming. It’s great to be able to work in an environment where exploration of new ideas and technologies are supported and encouraged. By operating the Bazaarvoice Labs team off-stack, it gives the Labs Engineers a chance not only to give input into what new products get built but what technologies get used to build them in a very low risk way.

The Tools We Use to Innovate in Bazaarvoice Labs (Part 1)

Hi everyone! This is my first post to the Bazaarvoice Developer blog and I’d like to take this opportunity to shed some light on some of the tools Bazaarvoice Labs has recently found very useful in creating the pilots and prototypes that ultimately morph into new products and features on the Bazaarvoice platform. Before I talk about our toolset though, I’d like to give you a quick rundown on what Bazaarvoice Labs is, our process and why it’s important for us to be flexible in our toolset choices.

Bazaarvoice Labs is the new product research and development group at Bazaarvoice with emphasis on the new and research. We are actually a team of engineers that report to our Product Management team (rather than through the engineering group) that help our Product Managers realize their wildest (and potentially most game-changing) ideas. Every quarter we evaluate and prioritize new ideas proposed by our Product Management team, customers and Bazaarvoicers around the company in order to research and create prototypes. The ideas we prioritize highest are those that come with big hairy assumptions but could change our business if they work. By building prototypes we’re able to suss out where the trouble might lie if we were to introduce the new product or feature to our entire customer base. We currently have over one thousand of the world’s biggest brands hosting their user-generated content in our platform and many large services organization to boot. The introduction of even a small new feature can have very large consequences to our organization. So on the risky stuff, we like to know where the gotchas lie. Some of the products spawned out of this process include BrandAnswers and Ratings and Reviews for Facebook (part of our SocialConnect Suite).

In order to build a prototype, we assign an engineer to work directly with a Product Manager or Product Designer. These two work together in an agile manner (agile with a little-a, not a capital A) in order to create a tangible prototype that demonstrates the Product Manager’s idea unencumbered by writing lots of requirements or unnecessary process. It’s this one-to-one relationship that makes this process hum, gives the creative process a kick in the pants and really lets these ideas properly gestate. Once more people get involved in the project, due to network effect, managing the project gets exponentially harder with each person you add and the need for process increases as a way to mitigate risk. By imposing a one-to-one structure for our prototyping teams we strip away any unnecessary obstacles to creativity and give real creative ownership to our Product Managers, Designers and Engineers. In a way, these teams become entrepreneurial cofounders as they attempt to prove their ideas. Additionally, by artificially constraining the initial project team to one engineer, the team is focused on building out the Minimally Viable Product needed to prove their assumptions and build a business case before further investment is required. Another nice side effect of this style of working is that it allows the engineer working on the project to choose their own tool-chain for each new project. Since they’re working alone on a project, there’s no need to constrain the tool choices to lowest common denominator of what every team member might already know or be familiar with. Of course, it’s up to the engineer’s discretion to reuse code or tools that may already be in use at Bazaarvoice but that choice ultimately lies with the engineer and the engineer knows to optimize around speed of creation vs. other organizational considerations. One nice side effect of an engineer being able to choose a new tool-chain with every project is that, in addition to proving business and product ideas, emerging technologies can be realistically evaluated and, where appropriate, integrated into our core engineering stack (this happened with requireJS which has become an integral part of how we deploy Javascript on our customer’s sites).

Sometimes simply building a prototype may not answer the questions that we have around the viability of the product and we need to take further steps to answer the questions we might have. For example, we needed to answer the following question for Ratings and Reviews for Facebook: “Will people be willing to read and write Product reviews inside a Facebook app?” In this case, a prototype isn’t enough. We needed to progress to the next phase of the process and actually pilot the application we had built with a couple of customers. For this reason, the “prototypes” we build need to be more robust than what you might initially think. Yes, we’re building concept cars in Labs but our concept cars actually need to run. We generally launch these pilots with three to five customers and generally won’t internationalize them. These restrictions keep us agile and help make sure we don’t have to build too much customizability into the pilots. Even though these pilots will only be launched with a handful of customers, some of these will be placed in some very high profile, high traffic places (some getting over 100,000 hits per day). Examples of running pilots right now include Nikon’s Ask and Answer for Facebook, Travelocity’s Social Connect Discovery Pilot (see the “Traveler Reviews” link) and TurboTax’s People Like You review search tool. Of course, when we launch pilots we track the data and rapidly move to improve the product and build out a suitable business case for productization. In the pilot phase the engineers are free to launch new code whenever they choose and must play the roles of UX, server side and operations engineer. By being chief cook and bottle washer on these projects, it frees the owning engineer (there’s still only one per project) to push releases as frequently as necessary to build that business case and observe how changes affect the project’s KPIs.

So what tools do we use to build software in Labs? Let’s review the two phases of our projects momentarily: The tools we chose to build with in Bazaarvoice Labs must support two phases of a project:

Prototyping: When the engineer needs to build a usable, tangible artifact targeted for internal consumption and demonstrations for clients.
Pilots: Where we launch our new ideas with a few select clients and measure results to build a business case. Pilots must be stable and scale yet the engineer still has to rapidly iterate on the feature set.

Because our development cycles are so short at Bazaarvoice, projects must also be able to transition between the prototype and pilot phase seamlessly. The tools we select must therefore support the requirements mentioned above. Generally we can divide our tool-chain into a few broad categories:

Operational Tools: Tools that help us keep things up and running
Server-side Application Development Environments: Application containers, full-stack and micro frameworks. Tools to build web apps with.
Data Storage and Management: SQL, noSQL and whatever else you need
Client-side Tools: Because there’s a lot you can do with just a browser nowadays
Measurement Tools: Without the data to back up our hypotheses, there’s no science

In my next blog post, I’m going to step through each of these categories and talk about a couple of tools that we use and the projects that we’ve used them in. This will not be an exhaustive list since we’re always evaluating new tools but it should give you some insight into the how and why we pick the tools that we do.

keep up the great piece of work, I read few posts on this site and I conceive that your web site is very ineirtsteng and has got circles of wonderful information.Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!

Listening for Search Improvements @ Lucene Revolution 2011

At Bazaarvoice, our business is powered by semi-structured data, and our mission is to unlock the potential of that data by delivering products and their reviews or answers to product-related questions, which are relevant to our clients and their consumers. Whether a consumer is trying to recommend a product that will answer another consumer’s question, or a merchandiser is trying to analyze reviews of the latest product launch, the vast majority of our platform functionality is powered by a search across a Solr/Lucene index. That’s why we were excited to attend the Lucene Revolution 2011 conference back in May 2011, and I wanted to share some key observations that can assist you in improving the search experience for your own users.

When integrating search into any application, you should first recognize that search across free-text or semi-structured data is unlike other features in your application in that the “correct” output for text search is often not well-defined. Typically, your requirements will state that searches should return “relevant” results, but for a non-trivial text corpus and an unbounded user query space, it is effectively impossible to define “relevant” for an arbitrary user query. In the face of this uncertainty, we developers tend to implement search in the manner we are accustomed for other features — configuring indexing and querying in a way that makes sense for a handful of cases we can imagine, and then checking the overall outcome for a dozen or so sample queries that we expect our users might enter. Once we are seeing reasonable results for our sample queries, and seeing no other ways to improve results across-the-board, we stamp the functionality as complete, and we move on to the next task, right?

The truth is that the lack of well-defined “correct” output for text search is actually the starting point for the implementation of another process — listening to what your end users expect from search. Unless you are the sole user, you as developer likely have only the vaguest understanding of how your users actually search. This is not a critique of you — it’s because every individual has developed their own process for formulating the terms they enter into that free-text search box you’ve provided. Fortunately, there are a number of common techniques for understanding user search behavior, which Otis Gospodnetić of Sematext outlined in his talk, Search Analytics: What? Why? How? Akin to web analytics, Otis described a number of key reports to use in measuring how well search is meeting the needs of end users. Among these are the top queries with zero results, top queries with the highest exit rate, words per query, and the top queries overall. Each of these reports can generally be created from query logs alone, and they are important barometers for evaluating and tuning the effectiveness of your search function. Using query logs, you can gauge the potential benefit of adding spell-checking (to address zero hits due to misspellings), query auto-completion (to assist with long queries), and hit highlighting (to see why results were considered relevant). After deeper query log analysis, you may even decide to preprocess user queries in a tailored way, as Floyd Morgan of Intuit described how they distill variations of the same query (e.g. “How do I input my 1099” and “Where do I enter a 1099”) to simpler customized search terms (“wheretoent 1099”) that provide better precision. As you can see, you can gain a significant understanding of your users’ expectations from query logs alone, but they do not provide the whole picture…

For even better analysis and performance of your search functionality, you need to pair query log data with some other form of feedback on result quality, usually click-through rates (CTR) on search results. Again, Otis described a number of metrics to compute when query logs are paired with CTR data. In a separate session, Andrzej Białecki of Lucid Imagination described how to take this data a step further, treating CTR as direct feedback on result relevance, and incorporating it into the actual document score. At first blush, this seems to be an ideal and straightforward search improvement, but Andrzej also identified a number of undesired effects that require conscious effort to avoid. Also, he highlighted that Solr/Lucene does not currently provide a preferred storage model for this type of data. Solr 3.x provides an ExternalFileField that is currently the best mechanism for incorporating document popularity based on CTR, while Lucene 4.0 is slated to deliver an efficient mechanism for storing popularity and other per-document values, which Simon Willnauer, a core Lucene committer, described in his session on Column Stride Fields a.k.a. DocValues. Finally, Timothy Potter of NREL described complementary techniques for improving result relevance by Boosting Documents by Recency, Popularity, and User Preferences. Obviously, every technique is not directly applicable to every scenario, but the approaches are common to many search applications, so it is worth considering how they could apply to your application.

As you can see, integrating search into an application is not a write-once, run-forever task. There is a wealth of opportunity for improving search so that it meets the actual needs of your users, and most of the information necessary for learning how to improve can be obtained by simply listening to your users using common techniques. So, I highly encourage you to review the sessions I have highlighted, and you can check out slides or videos for all the rest at the Lucene Revolution 2011 recap. Enjoy!

RHoK’ing Hackathon coming to Austin on December 2-4

In June, I travelled to Seattle to participate in Random Hacks of Kindness #3 (RHoK for short, which is typically pronounced “rock”). RHoK is a hackathon like many others, a chance to develop on a project for 24-36 hours and show off your application to the other hackers, but RHoK applies a slight twist to the traditional hackathon. Instead of trying to launch a new company in the span of three days or being confined to one particular API, RHoK aims to produce new ideas and solutions for social good, specfically producing software (and hardware) that will help those in crisis situations or dealing with climate change. RHoK is a 2-year-old event, holding worldwide coordinated hackathons at over 20 sites over the same weekend. Historically locations like Toronto, Bangladesh, New York City, and many others have hosted RHoK. RHoK Seattle was graciously hosted by Microsoft who allowed us to use several large conference rooms on their Redmond campus throughout the course of the weekend.

At the kickoff reception representatives of NASA, CrisisCommons, Microsoft and others talked about the need for tools to help first responders communicate effectively and applications to support those faced with a crisis (for example floods or earthquakes). Additionally, team members from NASA talked about the Open Government initiatives, which have produced large amounts of freely available data as well as the desire for NASA to produce more open source projects from our government. The reception was a great chance to meet other hackers from the Seattle area, along with a number of people who came in from Portland and California as we mingled and listened to music by DJ Maxx Destruct.

Starting with coffee and bagels on Saturday morning, we went through an exercise to determine the available skills of everyone at RHoK Seattle and then moved to start identifying the needs of the various problem definitions provided by the RHoK core team. There were representatives available for several of the problem definitions and the room gradually coalesced into a half dozen teams, each focused on a different problem and solution. The problems attacked at RHoK Seattle included the creation of mobile sensors that can easily be deployed around a disaster to capture environmental information at a variety of altitudes, real-time mapping of tweets and other data coming from a significant event, a notification application that pushes needs and directions to first responders via SMS, as well as a solution for connecting businesses with left over food with volunteers willing to deliver the food to those in need.

Just before lunch on Saturday, each team reported on exactly what they were intending to work on and what needs they had that other teams might be able to lend a hand on. Design and coding commenced, and by dinner many teams were able to stand and report on the functionality they had completed thus far. Many of the teams worked late into the night, and some even worked through the night. On Sunday the coding continued and at 4 pm each team took a turn presenting their work and talking about what they would do in the future. A panel of judges evaluated each presentation and the work of each team on its ingenuity, completeness, and a number of other factors and awarded prizes to the top teams. For a quick taste of RHoK Seattle, Johnny Diggz put together a video showcasing the weekend.

Having attended RHoK #3 in Seattle to see what the event is like, a few of our Engineers petitioned Bazaarvoice to sponsor RHoK in Austin. Now six months later: rhokaustin.org is ready to go. All the details for the Austin hackathon on December 2-4 are available on the site, and we’d love to have even more engineers, designers, HTML gurus, and project managers come to RHoK #4 in Austin. It’s going to be an amazing event and we hope to see you there!

Fixing a performance problem a few days before release (Part 2)

In the previous post, we discussed a performance issue we were facing, and some of the steps we tried while investigating it to no avail.

Profiling Concurrent Requests

Running concurrent requests a few times, I could indeed reproduce the 3x slowdown for new code that we were seeing in the performance environment. The next step was to profile each version during the concurrent test run.

Using the default JProfiler settings, the server was only able to process a few requests per minute. That wasn’t enough to see the performance effects, plus it was clear that profiling itself was taking far too much time and would skew the results considerably. After talking to a few folks, I was shown the CPU profiling options. There are two primary types of CPU profiling: Dynamic Instrumentation and Sampling.

profiling_concurrent_1

Dynamic Instrumentation, as the name implies, instruments all classes as they are loaded by the VM. Every method called will log data into JProfiler, making really small and fast methods appear much more expensive than they truly are. This can skew the results considerably for small, fast methods that are called many many times.

Sampling takes a sample of the JVM state every few milliseconds, thereby storing which methods are executing at any given time and extrapolating how long each method is taking overall. This method doesn’t allow us to know overall method counts, and therefore we cannot know the average amount of time per method call. However, it takes far fewer resources than instrumentation, especially for smaller methods.

After switching the setting, the server started up much faster during profiling and the number of requests handled while profiling was much more reasonable. Looking at the individual results after several responses were processed in parallel, the individual method that was causing the most processor time was quite obvious:

profiling_concurrent_2

The first method in the list, using 42% of the processing time, definitely appeared to be the culprit. To determine the specific method, JProfiler normally indicates the class name along with the method name. However, the class name for the method was incorrect: JProfiler thought that the method was in a JVM Generated Method Accessor. I haven’t determined the cause of this problem, but the offending method is primarily called by one other method, so it was easy enough to find.

Taking the wrong path

The method itself should have been very simple: it simply returned the product ID from an XML object. Commenting out the call to this method and rerunning the tests confirmed that this was indeed the problem. Because there was no indication of where the actual problem was, I made a first guess that the problem had to do with XML processing, specifically XPath accessors. I attempted to fix this problem by using other methods of property retrieval, but nothing I tried solved the problem when running the concurrent test run.

I reran the profiler with the newest code to see if any new information would show up. It should be noted at this time that each test run takes several minutes to setup and execute, so it had been quite a few hours of trial and error so far. Running the application while profiling takes even longer, so I had avoided running it again up to this point.

After running the application again, the top hit was StringBuilder.append. That was an odd problem because that method wasn’t using StringBuilder at all. It did have a simple String concatenation, but that should be pretty small:

Assert.isNotNull(idAttribute, "Product is missing required 'id' attribute: " + product);

Behind the scenes, the Assert.isNotNull method is doing something like the following:

public static void isNotNull(Object value, String message) {
    if (value == null) {
        throw new IllegalArgumentException(message);
    }
}

The more interesting part is that, though idAttribute is null very rarely (and hopefully never), the String concatenation happens every single time the method is called, which is millions of times! Plus, the product variable is rather large, and its toString() method generates a large XML String from the data. This simple line turned out to be the entire slowdown of the app.

The quick solution since we were so close to release was to replace the line with the logic in the isNotNull method. Because the String concatenation was then guarded by the null check, it was never actually executed. In addition, we looked through the codebase for other similar concatenations and removed them in the same way.

The long-term solution was to change the Assert class methods so that they used MessageFormat:

public static void isNotNull(Object value, String message, Object... messageParams) {
    if (value == null) {
        throw new IllegalArgumentException(MessageFormat.format(message, messageParams));
    }
}

The isNotNull calls look like the following:

Assert.isNotNull(idAttribute, "Product is missing required "id" attribute: {0}", product);

The String concatenation and the product.toString() method never get called unless the assertion fails, which should be rare. MessageFormat.format is more expensive than a String concatenation, but since it is called so infrequently, the performance savings more than make up for the slight cost of each call. Plus, the code is very clear and much more expressive than even using the String concatenation method.

Lessons Learned

There were quite a few lessons learned throughout this process. The first and likely most important is that simple, seemingly innocuous code can sometimes have a dramatic impact on overall performance of the system. In addition, I (re)learned quite a bit about performance debugging in general, and learned a lot about the performance testing tools in particular.

Debugging performance problems tends to take quite a bit of time just because of the shear volume of data or requests that inevitably cause the problem to begin with. Getting reproduceable scenarios and tests that can produce the same results time and time again for comparison is an added challenge, but extremely important in order to determine where the true bottlenecks are. By replaying production request logs to simulate traffic, the requests are very reproduceable and much closer to real-world scenarios. Failing to get reproduceable scenarios will lengthen the entire process.

These types of problems don’t come up often, but when they do, it’s really important to get them identified and fixed quickly. Hopefully this experience will help other developers crack open any performance problems and come to a quick solution.

Fixing a performance problem a few days before release (Part 1)

The Problem

Every few weeks we release a major version of our software. Throughout the release, we have many processes in place to ensure we don’t cause regressions in our latest code. Since we serve up millions of pages each day, regressions in scalability are just as important as failures.

At code freeze, which is approximately one week prior to release, we start running performance tests. The tests themselves are pretty basic: we have scripts that hit a server running new code with a large number of concurrent requests. We then compare compare the results with a baseline computed by executing the same requests against a server running the previous version’s code on the same hardware and database. The results are usually within a reasonable margin of error. This time, however, the new code was only able to process about 1/3 of the requests that the old code could.

In our current product offering, we have two main parts of public access: content display and content submission. Either one of these pieces could have been the culprit since we typically run all types of requests through the system during the first run. A subsequent run of just display traffic proved that display was causing the majority of the slowdown, even without submissions added to the mix. Another key piece of information is that the database didn’t appear to be the culprit: the processors on the app server were pegged, and the database servers were handling queries as normal. One of the usual suspects had an alibi! Of course, we weren’t ruling out the possibility that the database was still an issue, or possibly even Solr.

Finding the Bottleneck

The results so far indicated that we would probably need to profile the app and not just look for bad queries. In order to find the offending code, most Java profilers must connect to the JVM in a way that gives them access to low-level data. We use JProfiler for these types of issues, and it integrates very well with IntelliJ IDEA, but that requires reproducing the issue locally: not always an easy thing to do when large systems with enormous databases are involved.

Running locally, I would get different performance results than running in the performance environment. In fact, sometimes my Mac Pro is faster than the performance environment. Therefore, I would need to run both new and old code on my box to see how they compare – and where the breaking point is.

We have a few options for local testing. The first option is to extract a large data set for use locally. This option means importing the data into a local MySQL DB and reindexing the content into Solr. Unfortunately this is a multi-hour process for a large set of data. In addition, the request logs span all traffic for a particular server, so that means a multi-GB data set. I decided to start the extraction and import process in case this was necessary, and move on to Option 2.

The next option is to attempt to access our lab databases. The databases in the lab don’t have as much data as the extractions, but they do have enough to do a quick-and-dirty performance test. The big issue is that the databases are on different hardware, so it was possible that they wouldn’t match perfectly. I decided to try it anyway, and as expected from the original performance tests, the new code performed about half as well as the old code for each request. I was able to see these results by just executing the requests serially, so I thought I had a good candidate for profiling.

Profiling, Take 1

Profiling a single request in the new version, I didn’t find any method call that really stood out:

The highest cost method above is only 4%, and it turns out to be in the HTML templating system we use. JProfiler has the ability to compare two snapshots, so I took a snapshot for the new code, and then reran the request for the old code while profiling. Comparing the two, we see that there is really no change between the two codebases for these calls:

It looks like Freemarker could be the culprit, but looking at the increasing and decreasing parts of the results, they seem to cancel each other out. This doesn’t make sense if all of our time is supposedly in Java.

DB slowdown?

Looking more closely at the JProfiler snapshot output, there’s an option to show which methods and threads are being filtered out. The threads in the output above were all in “Running” state. This makes sense if the problem is within running Java code, but what if the problem is in the database? In that case, our Java code would be waiting a lot. It turns out that JProfiler has a specific option to show the JDBC calls. Looking at the following output, we can see a few calls are taking 15% of the time while running with the new code:

The nice thing about saving off a snapshot for each of the code versions is that I can keep looking at the data long after the server has been shutdown. Looking at the comparison between new and old code again, we see that there are a few queries that are taking 10x longer to execute in the new system:

This is a problem, but at that point, I realized that this is not likely to be our main problem. Noticing the number of invocations, we don’t call these queries any more often in the new code than the old. Instead, the same exact queries are taking longer. This could mean that we changed the indexes on these tables (nope!), or there is much more data in the database (probably not the case), or (most likely) the hardware is affecting performance of the DB queries drastically.

Switching DBs

Back to square one. I needed to use the same DB hardware for these comparisons, because it skews the results too much, mainly because DB query time overshadows processing time. Also, since we weren’t seeing heavy load on the DB servers during these tests, we were still thinking that the problem was in the Java code.

At this point I didn’t yet have a local DB environment that was reliable, so I decided to run against the performance testing DBs directly. We had to shut down performance testing, but otherwise this was surprisingly easy. While testing it out, I ran into an unexpected bonus: both new and old code was able to run against the new schema! That meant truly testing against the exact same DB was possible.

I ran the serial tests again in both new and old code, and found that they were actually quite similar in performance. Profiling both versions confirmed this, as there were really no differences between the new and old code outside a normal margin of error. It definitely wasn’t taking 3x longer to process requests. However, there is a lot of local processor dead time when handling requests serially because there is a ton of time spent waiting for the DB. I wasn’t able to reproduce the negative results running requests one at a time.

In the conclusion of this post, we’ll get past the red herrings and drive to a solution.

Can’t get into your favorite conference? No problem. Just create your own!

One of the best things about working in Austin is the sheer number of great software companies in town. From web-based start-ups to established enterprise software giants, there is huge diversity in the locally available business models, technology stacks, market sectors, and experience levels. It’s a good bet that no matter what technology problem that you’re facing, a few Austin companies have already found, built, or bought a solution before you.

About a year ago, I was having dinner with a friend who happens to work at another local software company. Throughout dinner we discussed a couple of different infrastructure technologies that our teams had recently built, some problems we were facing, and we shared ideas that might help solve some of those problems. We each walked away not only with ideas that might help solve our upcoming challenges, but also with a list of things that had already been tried and abandoned. While we each have intellectual property that we aren’t at liberty to share, that line leaves a ton of room for technology and ideas that our companies don’t wish to protect as IP. The big remaining question was “How do we do this again, and how do we get more of our team members learning from each other?”

Based on that experience, a few months later Bazaarvoice hosted our first ‘tech talk’ for the outside community. We invited half a dozen other companies from around Austin to come and learn about our architecture and some technologies that we had built that might be useful to others. Specifically, our agenda for the night was:

1. An overview of Bazaarvoice for those in the audience that didn’t know us (5 min)

2. An overview of our application architecture to support scale and uptime (45 min)

3. An infrastructure technology to easily configure Spring for multiple environment types (20 min)

4. A technology to supporting backwards compatibility and versioning in a REST API (20 min)

In return, the attending companies agreed to host their own talks at some point in the future. Fast forward a year, and the group has heard from four different companies and we’ve added a few new teams to the roster. For the investment of presenting 90 minutes worth of material, our teams have all continued to learn from the cumulative successes and mistakes of the whole group.

The good news is that Austin is not unique. There are places all over the country with a rich software industry that can replicate this idea. Think of it as a way to open-source some of your good ideas without the overhead of starting a blog or supporting a true OSS project (though you might want to do those things too). If you do consider it (and I highly recommend it), here are some tips to help make your group successful:

1. You are who you work for

One of the keys for the group was to have participation be officially sanctioned by the companies in attendance. While I can attend a technology group as an individual, I can’t share any of the great things that I’m working on unless Bazaarvoice agrees that those ideas are not IP that we want to protect. When it comes to sharing the inside scoop about your team’s technology, you certainly don’t want any misunderstandings or gray areas about what is appropriate and what isn’t. This also made it easy to continue the group, as we had established up front that every company in attendance for the original Bazaarvoice talk was also willing to share in return.

2. Like building software, don’t over engineer it

Getting something up and running doesn’t take a lot of infrastructure. We organized the first few meetings via email with a representative from each company. Did we have a fancy logo? Nope. Did we have a website to manage attendees? Nope. Did we have a name for the group? Nope (although it has since been informally dubbed “Austin Scales”). Did we have a budget? Nope. We just invited everyone to our office and provided some frosty beverages (this last part is highly recommended). Then, I found a volunteer to present the next time and let them organize it. They did the same for the time after that. You can always add those other nice-to-haves once your group is successful.

3. The first rule of fight club is “No recruiting.” The second rule of fight club is “NO RECRUITING!”

While the companies in attendance may not be competing with each other on the business front, we are often at war when it comes to finding and recruiting top talent. No company will want to approve participation in the group if they believe it will result in their best engineers being lured away. Make sure that everyone in attendance is on the same page. If your company is hiring, leave it at the door.

Why Columns are Cool

Near real time analytics on a large data set = hard!

About two years ago we found ourselves adding a lot of analytical features to the Bazaarvoice Workbench, our client facing analytics product. We were implementing various new dashboards, reports, and alerts. Dashboards needed to be near real time, so we were forced to either find a way to run required SQL queries in real time, or pre-compute required data. As the main data store was MySQL, we found it impossible to aggregate large amounts of data on the fly in real time, so we were forced to pre-compute a lot of aggregates. Pre-computation happens to be error prone, so we eventually went looking for other solutions which would not require us to do as much pre-calculation.

As we looked at the queries which we were running to pre-compute aggregate data, we spotted a common theme, i.e. most of the queries filtered and aggregated data. In the select clause of the query there would normally be a small number of columns with most values being aggregates such as counts, sums and averages. So we went brainstorming what type of database would potentially be a better fit for such queries compared to a traditional row oriented relational DBMS.

We looked at a number of different data stores which could work for the problem we had. One of the alternatives considered was Apache SOLR which is a great NoSQL search server we had been using for a long time already with great success. SOLR supports filtering and faceting which allows us to implement most things we were doing with MySQL using SQL. However, there is quite a bit of dev work which needs to go into indexing data in SOLR and querying it.

Enter the world of column oriented databases

Imagine a database which stores the data by column rather than by row. For example, we might want to store the following employee data in a table:

ID, Name, Department, Salary

1, Chuck Norris, DEV, 10000

2, Gate Bills, SALES, 20000

3, Bart Simpson, HR, 15000

Traditional row oriented databases would store column values for each row sequentially, hence the data is stored like this:

1, Chuck Norris, DEV, 10000

2, Gate Bills, SALES, 20000

3, Bart Simpson, HR, 15000

A column oriented database serializes values of each column together:

1,2,3

Chuck Norris, Gate Bills, Bart Simpson

DEV, SALES, HR

10000, 20000, 15000

This approach has some advantages over row oriented databases:

A columnar database does not need to read the whole row when a query only needs a small subset of columns. This is a typical pattern you see in queries performing aggregations. Not having to read the whole row from disk has an advantage of minimized disk I/O which really matters since databases are normally disk I/O bound these days.
Another important factor is data compression. If data is stored by column, it’s normally possible to achieve high compression rate for each column. This in turn helps to minimize disk footprint of the database, so the disk I/O is minimized as well compared to non-compressed data bases. It often makes sense to take a hit decompressing the data on the fly compared to reading more data from disk.

Getting to know Infobright

One of the column oriented databases we evaluated is Infobright which is an open source database built on MySQL by changing the storage engine to be column oriented. Of course a significant change to storage layout also means that Infobright had to replace MySQL’s query execution engine with a custom engine capable of taking advantage of the columnar storage model. So, let’s jump straight to test results we got when evaluating Infobright using a data set with 100MM records in the main fact table.

As you can see, the average query execution time for analytical queries was 20x faster than MySQL’s.

Additionally, the disk footprint was over 10x smaller compared to MySQL due to data compression.

Also, we found that Infobright supported most of the SQL syntax we were already using in our queries including outer joins, sub-queries and unions. We found that a little tweaking was still required to make some queries perform well.

The benchmark numbers were impressive enough that we ended up using Infobright to implement near real time analytics in one of our products. It does a great job at calculating aggregates on the fly so we no longer needed to maintain as many aggregate tables (for instance, daily, and monthly aggregates).

Infobright – the secret sauce

What is the secret sauce in Infobright? First, its column oriented storage model which leads to smaller disk I/O. Second, its “knowledge grid” which is aggregate data Infobright calculates during data loading. Data is stored in 65K Data Packs. Data Pack nodes in the knowledge grid contain a set of statistics about the data that is stored in each of the Data Packs. For instance, Infobright can pre-calculate min, max, and avg value for each column in the pack during the load, as well as keep track of distinct values for columns with low cardinality. Such metadata can really help when executing a query since it’s possible to ignore data packs which have no data matching filter criteria. If a data pack can be ignored, there is no penalty associated with decompressing the data pack.

Compared to our MySQL implementation, Infobright eliminated the need to create and manage indexes, as well as to partition tables.

Infobright – Limitations

Infobright Community Edition is the open source version of Infobright. Unfortunetly, it does not support DML (inserts, updates, and deletes), so the only way to get data loaded is bulk loads using “LOAD DATA INFILE …” command. It’s still possible to append data to the table, however there is no way to update or delete existing data without having to re-load the table. For some types of data (such as log files or call detail records), this is not a significant issue since data is rarely edited or deleted. However, for other projects this limitation may be a show stopper unless the data set is small enough when it can be periodically re-loaded into Infobright.

Infobright chose to take the SMP (Symmetric MultiProcessing) approach, so there is no built-in support for MPP (Massively Parallel Processing). This certainly limits the size of the databases for which query performance would be acceptable. However, it’s possible to shard the data across multiple Infobright instances and then aggregate the results. Shard-Query is an open source project which makes it possible to query a data set partitioned across many database instances, and our test results confirm that this approach works really well.

Summary

Column oriented databases proved to be a great fit for analytical workloads. We managed to eliminate the need to maintain a large number of aggregate tables by migrating our data warehouse from MySQL to Infobright. This in turn let us support near real time visualizations in our analytics products.

Thanks, this is useful information about column-oriented databases.

bazaarvoice: engineering

The official blog of Bazaarvoice R&D