Pet Insurance Providers

Project Motivation

I left my full-time job in February 2024 to spend more time with family. I decided to try my hand as a freelance analyst instead of working on a permanent, full-time basis.

Initially, I turned to the popular freelance website Upwork. After navigating the endless list of “Build Large Language Model (LLM)” or “Develop AI Chatbot” postings for fixed rates less than the vanilla latte in my hand, I applied for and accepted my first contract. The client request was relatively straightforward and boiled down to helping them uncover differentiating characteristics across pet insurance providers.

As luck would have it, I was also shopping for pet insurance for my almost two-year-old black lab named Falco (adopted name so jury is still out whether he was named after Keanu Reave’s character in the Replacements, the Austrian musician from the 80s (“Rock Me Amadeus” for any fellow members of the MTV generation), or the character from the Nintendo Super Smash Bros. series).

The company I left provided an embarrassment of riches of employee perks and subsidized pet insurance is one of many I was fortunate enough to take advantage of. Falco’s coverage would terminate shortly after my departure and as a result, this proved to be a great opportunity to inform my decision in tandem with assisting the client.

Web Scraping

Providers: 10

Reviews: 26,878

provider	reviews	date (min)	date (max)
AKC	838	01/24/2018	03/08/2024
ASPCA	5,556	03/21/2016	04/03/2024
Embrace	3,495	07/09/2019	04/03/2024
Fetch	2,049	01/06/2022	04/03/2024
Figo	1,803	12/18/2015	04/03/2024
Healthy Paws	2,211	02/23/2015	04/05/2024
Lemonade	1,302	09/01/2017	04/03/2024
Pets Best	4,279	12/14/2015	04/03/2024
Pumpkin	998	07/13/2021	04/03/2024
Spot	4,347	11/13/2019	04/04/2024

Sentiment Analysis

Sentiment Analysis performed using two popular Python libraries:

TextBlob
Vader

Model Evaluation

How much trust can we place in the results above?

In other words, how do we evaluate the efficacy of each approach for classifying sentiment?

Fortunately, the Trustpilot reviews contain star ratings, ranging from 1-star (lowest) to 5-star (highest)

To answer these questions, I treated the star ratings as expected sentiment and assigned as follows:

>= 4 (positive)
== 3 (neutral)
<= 2 (negative)

I viewed expected sentiment as the outcome variable and TextBlob and Vader sentiment assignments as the predictor variables.

Regressing expected sentiment on TextBlob and Vader sentiment assignments, and generating classification reports (below), is one way to evaluate the efficacy of each approach.

TextBlob Classification Report

provider	precision	recall	f1-score	support
AKC	0.92	0.92	0.92	723
ASPCA	0.92	0.90	0.91	4,678
Embrace	0.86	0.91	0.89	2,738
Fetch	0.90	0.92	0.91	1,699
Figo	0.86	0.92	0.89	1,368
Healthy Paws	0.90	0.99	0.94	1,891
Lemonade	0.90	0.95	0.92	1,064
Pets Best	0.83	0.95	0.89	3,209
Pumpkin	0.94	0.91	0.93	894
Spot	0.94	0.92	0.93	3,835

Vader Classification Report

provider	precision	recall	f1-score	support
AKC	0.94	0.87	0.90	723
ASPCA	0.94	0.88	0.91	4,678
Embrace	0.89	0.92	0.90	2,738
Fetch	0.92	0.90	0.91	1,699
Figo	0.89	0.89	0.89	1,368
Healthy Paws	0.92	0.97	0.95	1,891
Lemonade	0.93	0.92	0.92	1,064
Pets Best	0.85	0.92	0.88	3,209
Pumpkin	0.96	0.89	0.92	894
Spot	0.95	0.87	0.91	3,835

For the purpose of analyzing positive reviews, I prioritized Precision metrics over Recall; the subsequent analysis is based on the subset of data with a ‘positive’ Vader sentiment assignment

Positive Reviews Analysis

Approach:

Text pre-processing:
- Tokenization
- Stop words removal
- Stemming and Lemmatization

Bag of Words (BoW) model creation:
- Ended up with a prospective list of ~294,000 individual words or n-grams

Data reduction:
- In collaboration with the client, reduced list to ~280 words or n-grams, prioritizing focus areas including:
  - Ease
  - Efficiency
  - Service & Support

Independence Testing:
- Generated observed and expected counts for each provider/word (or n-gram) combination (ex. AKC/afford)
- Restricted analysis to provider/word (or n-gram) combinations with observed AND expected counts greater than 5
- Further restricted analysis to combinations with sufficient counts across all 10 providers
- Calculate Chi-square test statistic for each combination
- Calculate degrees of freedom and critical value
- Restrict analysis to combinations where Chi-square test statistic is greater than critical value
- Calculate odds ratio and 95% confidence interval for each combination

Note: The above approach yielded ~60 word/n-grams for the purposes of understanding differences in sentiment across providers.

“Ease” (Sample Reviews):

“I really love it because I can take my dog anywhere I want. It makes life much easier.”

“A very simple and easy experience.”

“Great coverage. Easy to file a claim. Quick pay back”

text	AKC	ASPCA	Embrace	Fetch	Figo	Healthy Paws	Lemonade	Pets Best	Pumpkin	Spot
claim easy	0.39	0.45	1.38	0.82	0.74	1.40	1.35	1.15	0.63	1.46
ease	0.43	0.72	1.27	0.87	0.95	1.54	0.70	0.87	1.00	1.29
easier	0.50	0.58	1.33	1.21	1.10	2.96	1.22	1.04	0.73	0.35
easy	0.37	0.89	1.21	0.77	1.01	0.94	1.71	0.76	0.84	1.50
easy file	0.67	0.31	1.34	0.82	1.41	1.25	1.92	1.43	0.58	1.03
easy file claim	0.87	0.33	1.35	0.89	1.44	1.09	2.04	1.26	0.61	1.08
easy submit	0.54	0.45	1.87	0.98	0.61	1.08	0.50	1.41	0.49	1.37
easy submit claim	0.64	0.45	1.90	0.99	0.58	1.13	0.55	1.33	0.59	1.30
easy understand	0.79	1.61	0.36	0.58	1.25	0.66	0.94	0.99	1.28	1.25
easy use	0.51	0.68	1.21	1.12	1.39	1.09	1.60	0.59	0.69	1.41
easy work	1.07	0.51	1.53	1.10	1.17	1.71	1.21	1.02	0.83	0.71
process easy	0.49	0.58	1.26	1.04	0.73	1.09	1.62	1.23	0.84	1.14
simple	0.32	0.55	1.27	0.78	1.27	2.18	1.95	0.82	0.52	0.93

“Efficiency” (Sample Reviews):

“Handle my claim quickly Very happy with your insurance”

“Fast and efficient reimbursement! Outstanding and caring customer service!”

“Prompt responses to claims and courteous staff.”

text	AKC	ASPCA	Embrace	Fetch	Figo	Healthy Paws	Lemonade	Pets Best	Pumpkin	Spot
claim quickly	0.91	0.55	1.27	0.68	0.69	3.22	0.43	1.24	0.80	0.69
efficient	0.76	0.54	1.36	1.00	1.01	1.61	2.26	0.95	0.69	0.79
fast	0.94	0.52	0.98	0.66	1.20	1.28	2.58	0.87	0.93	1.34
fast easy	0.86	0.65	0.97	0.45	1.37	0.88	2.66	0.64	1.27	1.51
prompt	2.23	0.66	1.01	0.72	0.78	1.28	0.85	1.50	0.92	0.91
promptly	1.80	0.41	1.60	1.53	0.93	2.18	0.57	1.18	0.93	0.41
quick	0.78	0.50	1.02	0.77	1.39	1.33	1.83	0.99	1.05	1.27
quickly	0.68	0.49	1.25	1.17	1.00	2.41	0.95	1.07	0.83	0.82

“Service & Support” (Sample Reviews):

“Years of excellent service great Company!”

“Wonderful professional experience from this company. Excellent and caring.”

“The staff is very knowledgeable and helpful with my questions.”

text	AKC	ASPCA	Embrace	Fetch	Figo	Healthy Paws	Lemonade	Pets Best	Pumpkin	Spot
caring	0.60	1.19	0.90	1.20	0.60	4.23	0.32	0.87	0.53	0.21
clear	0.48	0.78	0.96	1.34	0.90	1.92	1.25	1.12	0.71	0.71
courteous	2.48	1.40	1.43	1.04	1.32	0.72	0.80	0.74	1.45	0.26
excellent service	4.58	1.16	0.86	1.10	0.67	0.43	1.43	1.04	0.78	0.61
fair	1.71	0.78	1.03	0.83	0.83	1.45	0.90	1.78	0.44	0.60
friendly	0.71	0.90	1.01	1.24	1.29	0.90	1.98	0.68	1.37	0.92
great customer service	0.85	0.77	0.91	1.15	1.61	0.89	0.77	0.89	1.38	1.22
helped	1.39	0.64	0.84	1.94	1.07	2.32	0.91	0.88	0.65	0.62
helpful	1.22	1.04	0.88	2.02	1.15	0.73	1.00	0.72	1.43	0.83
kind	0.73	0.88	1.04	1.77	0.85	1.92	1.04	0.78	1.06	0.58
knowledgeable	1.44	0.96	1.31	3.58	1.00	0.32	0.34	0.57	2.33	0.46
nice	0.78	1.37	0.81	0.99	0.69	1.33	0.83	0.92	1.15	0.84
pleasant	1.33	1.19	1.17	2.31	0.92	0.71	0.73	0.59	1.45	0.60
professional	1.90	0.83	1.53	2.00	0.77	0.84	1.03	0.80	1.11	0.55
responsive	1.23	0.34	1.17	1.03	1.53	2.01	1.82	1.04	0.74	0.74
understanding	1.36	0.74	1.11	3.02	1.19	0.98	0.98	1.02	1.06	0.28