Executive Team Places Analytics Department on 90-Day Probation: Business Strategy at Stake
ANLY FPX 5510 Assessment 2 The executive team has been relying on business-critical decisions for the last few months based on figures reported by the analytics department. Some of these decisions have paid off, but others have cost millions. The analytics department, therefore, has been put on a 90-day probationary period. Once this period ends, the decision of whether the team remains or is dissolved will be made by the company management. In the interim, the head of the marketing department has been tasked with monitoring the performance of the analytics team, and all work will be thoroughly vetted before being brought before the managers for ultimate approval.
Business Problem: Evaluating Revenue through A/B Testing
In order to determine business performance, the company has selected A/B testing as one of its measures. This is done through measuring revenue generated per visitor from the same advertisement banner placed on two websites—Yahoo and CNN. The intention is to establish which website has more revenue from the firm’s advertisements. This measure is meant to simplify advert efforts and limit the use of third-party service providers.
There were two hypotheses in this A/B test. The null hypothesis was that Yahoo and CNN’s average purchase price were the same, and the alternative hypothesis was that there would be a huge difference in the purchase values on both websites. The test indicated that there was a huge difference in the purchasing behavior of customers, with Yahoo customers reacting more positively to the ad compared to CNN customers.
Testing Methodology: Applying Hypothesis Testing to Business Strategy
The testing method used is best aligned with the whole business issue. In the same way that A/B testing measures ad performance across two websites, the same method will be applied to measure the performance of the analytics team on probation. The objective is to identify the value of the team by how well they can provide good, actionable information.
During the 90-day probation, the analytics team will be exposed to a proper A/B testing process. The head of marketing will make two hypotheses: null hypothesis that the team will provide acceptable deliverables and alternative hypothesis that the team will perform poorly. The result will decide if the team stays as it is or gets dissolved.
A/B Testing: A Proven Marketing Experiment
| A/B Testing Process | Explanation |
| Hypothesis Formulation | Develop null and alternative hypotheses for the test. |
| Test Design | Create two versions of the content with one variable altered. |
| Data Collection | Show both versions to approximately equivalent sized samples of the target audience. |
| Analysis | Compare how each version performs to identify the best performer. |
ANLY FPX 5510 Assessment 2 A/B testing is a great marketing tool to compare campaign differences to identify which version performs best. For example, one segment of the audience reads version A of an advertising element and the other segment reads version B. The aim is to determine what material gives the best response. To create a successful A/B test, companies need to come up with two versions of similar content but with different single elements and then see how these two perform on two groups of audiences (Kolowich, 2019).
Requirements for Implementing A/B Testing in Business
| A/B Testing Requirement | Details |
| Research | Conduct research on past performance and determine the variables. |
| Development of Hypotheses | Develop hypotheses regarding future performance based on past data. |
| Test Implementation | Develop variations and implement the test for a sufficiently long duration. |
| Data Analysis | Provide proper representation and ensure no biased sampling or length pollution. |
Adequate research and hypothesis development should precede proper A/B testing. The research must focus on accuracy and correctness of past analytics reports to develop hypotheses on the basis of future performance. Having developed the hypotheses, the second stage is to develop a version of the control version and execute the A/B test. Executing the A/B test within the 90-day probation will enable the company to identify whether the analytics team can make a positive contribution towards the strategic aims of the company.
Yet, such possible issues as biased sampling and length pollution are likely to occur. Biased sampling happens when the test sample fails to represent the whole population, which can provide false results (Saleh, 2019). Length pollution occurs when an A/B test is terminated prematurely, the result being not enough data. For such high-traffic sites as Yahoo and CNN, testing for at least two weeks guarantees more trustworthy results (Saleh, 2019).
Addressing Potential Issues in A/B Testing
| Problem | Solution |
| Biased Sampling | Conduct tests regularly over one or two business cycles. |
| Length Pollution | Set the test length to obtain enough data. |
| Rater Bias | Employ objective metrics and provide bias-reduction training (Joseph, 2019). |
To fight against problems such as biased sampling, companies should conduct tests over several business cycles so that they have proper representation. To avoid contaminating with length, it is necessary to determine the right time to get enough data before closing the test. For top traffic sites, using a conversion minimum (say, 1,000 conversions) will help results be more powerful. Goal performance metrics will also keep rater bias away, one that could skew measures of the performance of the analytics team (Joseph, 2019).
Conclusion: Deciding the Future of the Analytics Department
The future 90 days will be crucial for the analytics department since their performance will come under scrutiny by following an A/B testing framework. The executive leadership will determine if they need to keep or close down the department based on the outcome. The work done by the team will be closely reviewed by the marketing manager to see if it meets the company’s standards of quality and accuracy. The probationary period will provide an indication of the potential contribution of the team to the company’s strategic plan.
References
Joseph, C. (n.d.). Problems With Performance Evaluations. Small Business – Chron.com. Retrieved from http://smallbusiness.chron.com/problems-performance-evaluations-1256.html
Kolowich, L. (2019). How to Do A/B Testing: A Checklist You’ll Want to Bookmark. Retrieved fromhttps://blog.hubspot.com/marketing/how-to-do-a-b-testing
Saleh, K. (2019). AB Testing: 14 Sampling Issues That Can Ruin Your Test. Retrieved fromhttps://www.invespcro.com/blog/ab-testing-14-sampling-issues-that-can-ruin-your-test/
Wingify (2019). What Is A/B Testing? Retrieved from https://vwo.com/ab-testing/