Tag Archives: integration

Maintaining Test Data with the “someObject” Test Structure

Language: Scala
TestTool: Scalatest

How did we get here?

When systems become reasonably complex, tests must manage cumbersome amounts of data. A test case that may test a small bit of functionality may start to require large amounts of domain knowledge about the system being tested. This is often done through the mock data used to set up the test. Maintenance of this data becomes cumbersome, monotonous and can feel Sisyphean. To solve these problems we created “someObject”, a modular system that allows us to maintain data in only one location while providing the flexibility to create specific data for our tests.

Let’s Do A Code Time!

To start this post, we’re going to build a system without the “someObject” test structure to provide context for its use. (To skip to the “someObject” structure, jump to here!). Suppose we are building a service that reports on advertising campaigns. We may create a class that describes an advertising campaign and call it `Campaign`.

case class Campaign(id: String,
    name: String,
    client: String,
    startDate: UTCDate,
    endDate:UTCDate,
    deleted: Boolean)

Now we are going to store this campaign in a database, and we need to write some integration tests to make sure this operation is performed properly. A test that confirms a campaign is stored properly might look like this:

it should "properly store a single campaign" in {
    Given("we have a proper campaign")
    val campaignToStore = Campaign(
        id = "someId",
        name = "someName",
        client = "someClient",
        startDate = UTCDate(1955, 10, 6),
        endDate = UTCDate(1956, 10, 6),
        deleted = false)

    And("we store this campaign in the database")
    database.storeCampaign()

    When("we fetch the given campaign")
    val fetchedCampaign:Campaign = database.fetchCampaign()

    Then("All the campaign values were stored properly")
    fetchedCampaign.id shouldBe campaignToStore.id
    fetchedCampaign.name shouldBe campaignToStore.name
    fetchedCampaign.client shouldBe campaignToStore.client
    fetchedCampaign.startDate shouldBe campaignToStore.startDate
    fetchedCampaign.endDate shouldBe campaignToStore.endDate
    fetchedCampaign.deleted shouldBe campaignToStore.deleted
}

BlogIntro.scala

Add some functionality:

Now we add the ability to update some values for this campaign in the database, and we need to test this new piece of functionality. That test might look something like this:

it should "properly update a single campaign" in {
    Given("we have a proper campaign")
    val campaignToStore = Campaign(
        id = "someId",
        name = "someName",
        client = "someClient",
        startDate = UTCDate(1955, 10, 6),
        endDate = UTCDate(1956, 10, 6),
        deleted = false)

    And("some update parameters for our campaign")
    val updateParameters = UpdateParameters(name = Some("someNewName"))

    And("we store this campaign in the database")
    database.storeCampaign(campaignToStore)

    When("we update this campaign")
    database.updateCampaign("someId", updateParameters)

    val fetchedCampaign:Campaign = database.fetchCampaign("someId")

    Then("All the campaign values were stored properly")
    //Unchanged values
    fetchedCampaign.id shouldBe campaignToStore.id
    fetchedCampaign.client shouldBe campaignToStore.client
    fetchedCampaign.startDate shouldBe campaignToStore.startDate
    fetchedCampaign.endDate shouldBe campaignToStore.endDate
    fetchedCampaign.deleted shouldBe campaignToStore.deleted
    //Changed values
    fetchedCampaign.name shouldBe updateParameters.name
}

BlogUpdateFunction.scala

But here we see the duplication of test boilerplate code in `campaignToStore`. We don’t want to have to copy over `campaignToStore` into every test, so we might want to abstract that out to be used all over the suite.

object MyCampaignTestObjects {
    val campaignToStore = Campaign(
        id = "someId",
        name = "someName",
        client = "someClient",
        startDate = UTCDate(1955, 10, 6),
        endDate = UTCDate(1956, 10, 6),
        deleted = false)
}

BlogAbstractToSuite.scala

Now we can use the same data in every test!

Add a Test that requires unique data:

Suppose we now write a function to fetch all the campaigns that are stored in the database. We might need a test that involves uniqueness in the data we store, such as the following example:

it should "properly fetch all stored campaigns" in {
    Given("we store several unique campaigns")
    val anotherCampaignToStore = Campaign(
        id = "someSecondId",
        name = "someSecondName",
        client = "someSecondClient",
        startDate = UTCDate(1955, 10, 6),
        endDate = UTCDate(1956, 10, 6),
        deleted = false)
    database.storeCampaign(campaignToStore)
    database.storeCampaign(anotherCampaignToStore)

    When("we fetch all campaigns")
    val allCampaigns = database.fetchAllCampaigns()

    Then("all campaigns are returned")
    allCampaigns shouldBe List(campaignToStore, anotherCampaignToStore)
}

BlogTestWithUniqueTestData.scala

In the example, we re-used the campaign we abstracted out earlier for conciseness, but this makes this test unclear that `anotherCampaignToStore` is unique. What if someone else comes in and changes `campaignToStore` and it happens to match data from `anotherCampaignToStore`? This test would then become flakey and nobody likes flakey tests. We might decide to just make all data used in this test local to this test, but then we will need to maintain the test data in both this test, and `MyCampaignTestObjects`.

Add Some Arbitrary Data Constraints:

Suppose now that there is a new design constraint on how campaigns can be stored in the database. Now all client names must be lowercased in all campaigns:

object MyCampaignTestObjects {
    val campaignToStore = Campaign(
    id = "someId",
    name = "someName",
    //We change the client name to match our new requirement
    client = "some_client",
    startDate = UTCDate(1955, 10, 6),
    endDate = UTCDate(1956, 10, 6),
    deleted = false)
}

BlogNewConstraint.scala

Now we start to see the issue with maintaining test data across the whole suite that we’ve been constructing. We need to find every mock campaign that is used in our suite and ensure that its client field data is lowercased. Realistically, many of our tests, (specifically in this example, the `fetchAllCampaigns` test) don’t care about the client field of their campaign, and so we shouldn’t need to care about the client field value while setting up our mock test data. Because this example is small, it’s not cumbersome to directly update the value to satisfy the new constraint. Now let us Imagine a large set of suites, each containing hundreds of unique test cases. Suddenly this single suite requires a large amount of work to refactor one field across each test case. Nobody wants to do that monotonous maintenance. To address this, our team adopted the “someObject” structure to minimize this data maintenance within our tests.

someObject Test Structure:

When designing this test structure, we wanted to make our test data extendable for use anywhere it is needed. We used Scala’s `trait` to mix in necessary functions to provide test objects to the objects inside our tests, such as the `MyCampaignTestObjects` object above:

trait CampaignTestObjects {
    def someCampaign(id: String = "someId",
                     name: String = "someName",
                     client: String = "some_client",
                     startDate: UTCDate = UTCDate(1955, 10, 6),
                     endDate: UTCDate = UTCDate(1956, 10, 6),
                     deleted: Boolean = false): Campaign =
        Campaign(
            id = id,
            name = name,
            client = client,
            startDate = startDate,
            endDate = endDate,
            deleted = deleted)
}
 object MyCampaignTestObjects extends CampaignTestObjects {
    //Any other setup methods for test data
}

Now we can revisit our `fetchAllCampaigns` test example.

it should "properly fetch all stored campaigns" in {
    Given("we store several unique campaigns")
    val campaignToStore = someCampaign(id = "someId")
    val anotherCampaignToStore = someCampaign(id = "someNewId")
    database.storeCampaign(campaignToStore)
    database.storeCampaign(anotherCampaignToStore)

    When("we fetch all campaigns")
    val allCampaigns = database.fetchAllCampaigns()

    Then("all campaigns are returned")
    allCampaigns shouldBe List(campaignToStore, anotherCampaignToStore)
}

BlogBasicSomeObject.scala

Inside this test, we’ve set up two unique campaigns, by calling the `someCampaign` method from our test data trait. This populates the returned campaign with dummy data that we don’t care about. All we need out of this method is “some campaign” with “some data”. Now, instead of obscuring the intent of the test case by setting up cumbersome, overly-expressive mock data, we can simply override the implicitly available mock objects with only the necessary data. For the unique campaigns needed in the `fetchAllCampaigns` test, we only only really care about each campaign’s identifier. We don’t update the name, client, startDate, etc. because this test doesn’t care about any of that data. We only need to care that the campaigns are unique for our database. Under this test structure, when we receive the design change about the client names being lowercased, we don’t need to update our `fetchAllCampaigns`.

Another Example:

Let’s provide another example that our team encountered. Campaigns inside our database now need to also store the amount of money spent on each ad campaign. We’re adding a new column to our database, and changing the database schema.

case class Campaign(id: String,
    name: String,
    client: String,
    totalAdSpend: Int,
    startDate: UTCDate,
    endDate: UTCDate,
    deleted: Boolean)

Now, every test that has a campaign involved needs to be updated to include a new field; but under the “someObject” structure we only need to add two lines and all existing tests should be working fine again:

trait CampaignTestObjects {

    def someCampaign(id: String = "someId",
                     name: String = "someName",
                     client: String = "some_client",
                     totalAdSpend: Int = 123456,
                     startDate: UTCDate = UTCDate(1955, 10, 6),
                     endDate: UTCDate = UTCDate(1956, 10, 6),
                     deleted: Boolean = false): Campaign =
        Campaign(
            id = id,
            name = name,
            client = client,
            totalAdSpend = totalAdSpend,
            startDate = startDate,
            endDate = endDate,
            deleted = deleted)
}

BlogPostSchemaChange.scala

 

Behavior Driven Tests:

The purpose of the “someObject” structure is to minimize data maintenance within tests. We want to ensure that we’re disciplined about only setting data that the tests need to care about. There are cases where data might seem necessary for what we are testing, but we can abstract that data away to de-couple the test’s reliance on hard coded values. For example, suppose we have a function that returns the sum of all the `totalAdSpend` across our database.

def sumAllSpend(campaignsToSum: List[Campaign]):Int = 
    campaignsToSum.reduce(_.totalAdSpend + _.totalAdSpend)

To test this function, we might write a test like this:

it should "properly sum all totalAdSpend" in {
    Given("we store several unique campaigns")
    val campaignToStore = someCampaign(id = "someId", totalAdSpend = 123)
    val anotherCampaignToStore = someCampaign(id = "someNewId", totalAdSpend = 456)
    database.storeCampaign(campaignToStore)
    database.storeCampaign(anotherCampaignToStore)

    When("we sum all ad spend")
    val totalTotalAdSpend = sumAllSpend(database.fetchAllCampaigns())

    Then("the result is the sum of all ad spend")
    totalTotalAdSpend shouldBe 975
}

BlogPostNonBehaviorTest.scala

While this test does work, and it utilizes this “someObject” structure, it still forces data management at the test level.

`sumAllSpend` doesn’t really care about any one campaign’s `totalAdSpend` value. It only cares that we add all of the `totalAdSpend` values up correctly. We could instead write our test to assert on this behavior instead of doing the math ourselves and taking on the responsibility of managing more data.

it should "properly sum all totalAdSpend" in {
   Given("we store several unique campaigns")
   val campaignToStore = someCampaign(id = "someId")
   val anotherCampaignToStore = someCampaign(id = "someNewId")

   val allCampaignsToStore = List(campaignToStore,anotherCampaignToStore)
   allCampaignsToStore.foreach(campaign => database.storeCampaign(campaign))

   When("we sum all ad spend")
   val totalTotalAdSpend = sumAllSpend(database.fetchAllCampaigns())

   Then("the result is the sum of all ad spend")
   totalTotalAdSpend shouldBe allCampaignsToStore.reduce(_.totalAdSpend + _.totalAdSpend)
}

BlogPostBehaviorTest.scala

With this test, we don’t care what campaigns sales were, we don’t care how many campaigns are stored, and we don’t care about any constant value. This test will return the sum of all campaign’s `totalAdSpend` value that we store in the database.

Conclusion:

In this introductory blog post, we explored the someObject testing structure in scala, but this concept is not intended to be language specific. Scala makes this concept easy to implement through the use of Default Parameter Values but in a future post I’ll show how it can be implemented through the Builder Pattern in a language like Java. Another unexplored “someObject” concept is the granularity of control in setting default data. This post introduces the “global” and test specific setting of default data, but doesn’t explore how to set test suite level data for our test objects, and the cases in which that might be useful. I’ll discuss that the future post as well.

Code snippets:

BlogIntro.scala
BlogUpdateFunction.scala
BlogAbstractToSuite.scala
BlogTestWithUniqueTestData.scala
BlogNewConstraint.scala
BlogBasicSomeObject.scala
BlogPostSchemaChange.scala
BlogPostNonBehaviorTest.scala
BlogPostBehaviorTest.scala

Partner Integrations: Do’s and Don’ts

In this blog post , a Senior Product Manager on our Product team, discusses challenges to building and maintaining technical partnerships between organizations as well as provides advice on how to overcome those challenges.

Every company comes to a point, early or late, where it realizes that it must partner with other companies to drive value in the market. Partnerships always start with conversations, handshakes, and NDAs. At some point, unlocking the value of partnership may hinge upon establishing a formal integration between the two companies. These integrations constitute a technical “bridge” between companies. They can unlock otherwise inaccessible value, allow for one company to OEM the other, and/or can accelerate work that otherwise is “re-invented” each time the companies engage each other.

Integrations can be amazing vehicles to create value that only comes from combining capabilities from separate entities, while simultaneously allowing each entity to focus on what each one does best. They can be the perfect manifestation of the all too often promised “complimentary” value. Integrations can offer consistency, repeatability, and reduced friction in the activities involved in unlocking that value.

Unfortunately, integrations are often approached in manner in which all parties involved are not setup for success. Why?

Integrations aren’t just some “code.” They are product. They require an organized effort to build, including technical staff and non-technical staff to build (engineers, architects, project manager, product manager, partnership manager). They require support, assigned owners, subject matter experts, marketing, documentation, and proper roadmap vision. Integrations demand the same attention and focus that any first class “product” requires.

Integrations require both more and different types of communication. Because the value of the integration is typically not front-and-center to the core value of each org, there must be additional effort to communicate the existence of the integration and the value it brings within each org. Sales, onboarding support, post-live support organizations all need ways to communicate with the other integrated party (who calls who when something stops working?). The two product organizations must communicate ahead of any changes to dependent technologies such as APIs. A classic communication gap happens when one entity changes their APIs and doesn’t let the other party know soon enough or not at all. Problems are only discovered when something breaks.

Integrations are usually birthed by the wrong part of the org. The challenge with integrations is that the impetus to create them usually originates from one or both of the company’s business development/partnerships team – a group that typically has little appreciation for the discipline of product management. Their priority is on “relationships” that historically focus on non-technical efforts. Additionally, the ADD-like attention span of most partnerships teams results in a great desire to create an “integration” with a partner for marketing and sales-driven reasons, but very little attention, effort, and commitment to the long-term requirements of a properly supported product. It is quite easy to just stop communicating with a partner who is no longer deemed valuable, but such an about-face cannot be made when an integration is in place with paying customers. Most often, partnerships orgs do not have technical resources within their structure, but rather rely on borrowing technical resources from wherever they can be found (“Hey, I have a partner company who just trying to do this thing, and they have a quick technical question…”). This is a great approach for proof-of-concept initiatives, but certainly not for something that companies/customers must trust to deliver value. The product organizations at each company must be responsible for bringing an integration to life. Regardless of whether the product org has enough resources to service an integration like a first-class product citizen, at least the owner will have an understanding of what is and isn’t getting handled properly and can mitigate the potentially negative outcomes that arise from under-served products.

Correctly structured incentives are crucial to the short and long-term success of integrations. There must be something in it for all concerned parties. Direct compensation and rev share are two good options. You should be cautious of such benefits as “stickiness” (as in, the assumption that giving an integration free-of-charge to an existing customer makes that customer less likely to debook your core service) or the halo effect associated with integrating with a company (i.e. “Do you know we’re integrated with Facebook?”). Many integrations have been built on the promise of return. Once that promise begins to fade (from any one or more of the parties), so does the motivation of the affected party to keep up their end of the technical bargain. The technology world’s version of “he’s just not that into you (anymore).” Once an integration is no longer properly attended to from one party, the integration becomes a liability. It’s not enough for the bridge to be secured to just one side of the river.

People love to build stuff. But, they hate to support it. There must be something in it for the platform to properly prioritize integration maintenance efforts. Be wary of agreements that lack commitments, SLAs, etc. (often termed that they will do any needed work within the bounds of “best efforts”) as these agreements allow the company responsible for the integration (“code”) to elect to not invest in the support and roadmap development, should their interest wane. If the agreement lacks commitments, then the partnership will likely as well. They will acknowledge the maintenance effort, but it will always get pushed to the next dev cycle. Which leads us to…

The Challenge of Opportunity Cost

The assumption here is that these companies contemplating an integration are predominantly product organizations. Their company mandate is to bring products to market at scale. This is dramatically different than a service organization who essentially trades dollars for hours.
This means that the cost of the technical/engineering effort at a product organization is different than that of a service organization. Not in that engineers get paid more at product organizations, but rather the additional opportunity cost of engineering effort at a product organization often introduces an impossibly high hurdle rate for putting those engineers on non-core “integration work.” Even just the existence of opportunity cost, albeit uncalculated, is all that a dissenting product or engineering leader needs to de-prioritize seemingly less important “integration work” that doesn’t deliver core value.

One innovative approach to solve this dilemma is to use outsourced engineering resources from a service organization to avoid the challenges that comes with opportunity cost. It makes good business sense: let your in-house engineering staff concentrate on doing things that drive core value at scale. The downside of this approach is that there is a very clear and visible cost (hrs * hourly rate) that is attached to all effort associated with the integration. A similar cost analysis is rarely thought about when utilizing internal resources, so the integration product manager should be prepared. Getting things done is always more expensive than you thought.

Of course, another solution is to simply make integration work of the same perceived class of value as that of the core product org’s core solution. However, as we describe above, this can be a big challenge.

The technical approach must be at the convergence of correctly structured incentives and technical viability. How open or closed a platform is can dictate how an integration can be executed. The associated partnership incentive structure can dictate how an integration should be executed. The resulting integration will result from the intersection of these two perspectives.

Closed platforms force the work on that platform. Open platforms allow for more options – either or both entities, possibly even a third-party, can contribute to the integration development.

Let’s look at a few scenarios.

Scenario 1: B is a “closed” platform

b_is_closed_platform

“Closed” here means that the platform does not allow for integration (read: code) to be hosted by that platform and that the platform does not have externally accessible APIs to utilize from outside the platform. The closed platform may have internally accessible APIs, but those do an external party little good.

Closed platforms force that platform to do the integration work. Thus, there must be incentives for the closed platform to both build and support the integration long-term. The effort to build the integration is often simply the result of the opportunistic convergence of both parties being sold on (at least) the promise of value and some available engineering capacity within the close platform. Without the proper incentives for platform B, this becomes a classic example of the issue of the Challenge of Opportunity Cost, discussed above. The engineer who had some free time to build the integration is suddenly no longer available to fix a bug or build a new feature. There must be motivation in some form to continue to maintain the integrity of the integration.

Scenario 2: B is open

b_is_open_platform

Open platforms present more options. In scenario 2, B is no longer the only entity who can develop the integration. A, B, or a third-party org can build the integration. There are more alternative incentive structures as well. Since the engineering effort can be executed by a non-B entity, there doesn’t need to be much in it for B (there can be, but it is not near the necessity). There will certainly need to be knowledge of the B platform (documentation, sandboxes, API keys, deployment directions, etc.) on the part of the developing entity, but this effort on the part of B has a much lower hurdle rate than that which is required to get something into B’s engineering roadmap. Typically, B will have some form of partner “program” by where such assets and knowledge are available for a predetermined fee. Even in absence of such a program, the needs are significantly less than if the development effort required engineers from platform B to do the build work.

Scenario 3: Middle-ware Solution

middleware_solution

Scenario 3 is just a derivative of Scenario 2. Options are abundant. A, B, or a third-party can build the integration. In most cases, any of those entities can bring the integration to market. A major decision will be how and where to host the middle-ware solution and how to provide production-ready support, specifically beyond the initial build phase (which can just leverage cloud hosting services like Amazon, etc. to quickly get up and running). The trade-off is that such a middle-man solution removes any challenges that come with the need to host the integration within the B platform, which can range from simple plug-n-play effort to per-instance customizations required for each integration incarnation.

Incentive options are very similar to Scenario 2. One exception is that there is a clear opportunity for a third-party to bring the integration to market with an associated price tag.

Summary

Integrations are powerful and often hugely valuable, but their success is directly tied the ability to structure them for the long-term. Integrations are a special kind of “product” requiring different types of communication and can benefit from the use of outsourced resources to execute and maintain.

A successful integration is the result of a technical and non-technical relationship that is structured in a way that provides benefit to both parties that can adequately compensate for the often underestimated level of involvement required across both organizations.