# Mentioned Isn't Recommended: How ChatGPT and Gemini Recommend HR Software **CobaltKite Research 001 (CKR-001)** Version 1.0.0 · Data collected October 2026 Author: Tejas D Jaiprakash, CobaltKite ([ORCID 0009-0008-3884-4141](https://orcid.org/0009-0008-3884-4141)) Study page: https://cobaltkitecreative.com/research/mentioned-isnt-recommended/ DOI: https://doi.org/10.5281/zenodo.23361807 License: CC BY 4.0 --- ## Summary I asked ChatGPT and Gemini which HR software a 50-person US startup should buy. Three different prompts, ten runs each, on both assistants: 60 answers in total. Then I tracked two things for every vendor. Was it named at all, and did it make the top three of the recommendation list? Those two numbers often tell very different stories. - **Deel was named in 68% of answers (41 of 60) but made the top three in just 1 of 60.** - **Rippling was ranked first in 52 of 60 answers (87%)** and was in the top three of every single one. - **Four vendors (Rippling, Gusto, BambooHR and Justworks) took 99% of all top-three slots.** - **Workday was named in 13 of 60 answers and never recommended.** Every one of those mentions told the buyer not to use it at their size. Most AI visibility tracking counts how often a brand gets mentioned. This dataset shows why that number on its own can mislead. Being named is not the same as being recommended, and sometimes a mention is a warning. This repository holds every answer word for word, the coding behind every number, and the script that produces the results, so anyone can check the work or run it again. --- ## Why I ran this I run CobaltKite, an AEO (answer engine optimization) agency. A big part of the job is telling a company how visible it is inside AI assistants. The usual metric is mention rate: out of N answers, how many named you. I wanted to know if mention rate holds up once you look at what the answer actually does with the brand. So I picked one category, one buyer and a fixed protocol, and logged everything. --- ## What I tested ### The buyer and the prompts One buyer: a 50-person SaaS startup in the US. Three ways of asking, from casual to considered. Exact text is in `prompts.csv`. | ID | Style | Prompt | |---|---|---| | P1 | Quick search | best HR software for a 50 person startup in the US | | P2 | Problem-led | We're a 50 person SaaS startup in the US. Onboarding, time off and employee records are all in spreadsheets and it's getting messy. What HR software should we switch to? | | P3 | Shortlist | I'm evaluating HR platforms for our 50 person US SaaS startup. Which ones should be on my shortlist, and what is each one best for? | ### The assistants | | ChatGPT | Gemini | |---|---|---| | Model | GPT-5.6 Sol | Gemini 3.1 Pro | | Reasoning setting | Thinking: High | Extended thinking: off | | Chat mode | Temporary chat, unpersonalized | Temporary chat | | Personalization | Off | Off (personal context disabled) | | Web search used | 30 of 30 runs | 25 of 30 runs | This version covers ChatGPT and Gemini. Claude is planned for version 2.0. ### Protocol - 10 runs per prompt per assistant. 3 prompts x 2 assistants x 10 runs = 60 runs. - A fresh chat for every run. No follow-up messages. - Prompts rotated P1, P2, P3, then repeated, so no single prompt got bunched into one time of day. - Fresh browser profile, language set to English (US). - Web search was left at each assistant's default. I didn't force it on or off. - Runs were made from outside the US without a VPN. - No assistant asked a clarifying question in any run. - All runs were collected in October 2026. --- ## How the answers were coded Every answer was coded in two ways. **1. The recommendation list (coded by hand).** For each run I wrote down every vendor the answer put forward as an option, including "also consider" asides, in the order the answer ranked them. That raw list is in `runs.csv` as `coded_list_raw`. When scoring, plan and product names are folded into the vendor (Gusto Plus counts as Gusto, Deel HR Core as Deel, Justworks PEO as Justworks, and so on). One answer offered "Deel or Remote" as a single slot, so both vendors get that rank. **2. Mentions (detected automatically).** `analysis.py` searches each answer's text for a fixed list of HR, payroll, PEO and EOR vendors. Links are stripped first, so a URL never counts as a mention. Tools that are not HR software (Slack, Google Workspace, Okta) and review sites (G2, Capterra, Reddit) are not counted as brands. The full name list is at the top of `analysis.py`. The two can disagree, and that is the point. A brand can be named in passing ("skip enterprise suites like Workday") without ever being put forward. Those cases show up in `mentions.csv` as `mentioned = 1`, `listed = 0`. ### Definitions | Term | Meaning | |---|---| | Mentioned | The brand's name appears anywhere in the answer text. | | Listed | The brand appears in the coded recommendation list for that run. | | Position | The brand's rank in the coded list, starting at 1. | | Top 3 | Position 1, 2 or 3. | | Ranked #1 | Position 1. | --- ## Files | File | What it is | |---|---| | `README.md` | This document. | | `runs.csv` | One row per run (60 rows). Settings, the coded list and the full answer text. | | `mentions.csv` | One row per brand per run where the brand was mentioned or listed (325 rows). | | `prompts.csv` | The three prompts, verbatim. | | `summary.csv` | Every number behind every finding, with confidence intervals. | | `analysis.py` | Rebuilds `mentions.csv` and `summary.csv` from `runs.csv` and prints the results. | ### runs.csv | Column | Description | |---|---| | run_id | Unique ID, e.g. `CKR001-CG-P1-01` (CG = ChatGPT, GM = Gemini). | | platform | ChatGPT or Gemini. | | model_version | Model selected at the time of the run. | | reasoning_setting | Thinking or extended thinking setting used. | | prompt_id | P1, P2 or P3. See `prompts.csv`. | | run_in_cell | Run number (1 to 10) within that assistant and prompt. | | collection_period | When the data was collected (YYYY-MM). | | personalization | Account personalization state. Off for all runs. | | fresh_chat | Whether the run used a new chat. Yes for all runs. | | location_note | Where the runs were made from. | | web_search | Whether the assistant searched the web for that answer. | | asked_clarifying | Whether the assistant asked a question instead of answering. | | coded_list_raw | The hand-coded recommendation list, as logged. | | brands_listed_count | Number of distinct vendors in the coded list. | | brands_mentioned_count | Number of distinct vendors named anywhere in the answer. | | answer_text | The full answer, exactly as returned, including markdown and citation links. | ### mentions.csv | Column | Description | |---|---| | run_id, platform, prompt_id | Which run. | | brand | Vendor name. | | mentioned | 1 if named anywhere in the answer text. | | mentioned_outside_citation_links | 1 if still named after removing citation link labels. Used as a check. | | listed | 1 if in the coded recommendation list. | | position | Rank in the coded list. Blank if not listed. | | in_top3 | 1 if position is 1 to 3. | | first_mention_order | Order in which the brand first appears in the text, among detected brands. Used as a check. | ### prompts.csv `prompt_id`, `prompt_style`, `prompt_text`. ### summary.csv | Column | Description | |---|---| | finding_id | Which finding the row supports (see below). | | metric | What is being measured, e.g. `mentioned:Deel` or `top3:Rippling`. | | group | The slice: All, an assistant, a prompt, an assistant-prompt pair, or a leave-one-out slice such as `drop P2`. | | k, n | Count and total, where the metric is a rate. | | value | The result. | | ci_low, ci_high | Interval. The `method` column says what kind. | | method | How the interval was calculated. | Finding IDs: F0 brand rates in every slice · F1 Deel (F1-LOO for leave-one-out) · F2 Rippling at #1 (F2-LOO) · F3 top-four concentration · F4 brands per answer · F5 assistant vs wording · F6 Deel by assistant · F7 web search use · F8 BambooHR at #1 by prompt · F9 HiBob by assistant · F10 brands named but never listed. --- ## Statistics With 60 runs, the uncertainty matters as much as the numbers, so every rate in `summary.csv` comes with an interval. - **Rates** use Wilson 95% confidence intervals. - **Differences** use a bootstrap: runs are resampled within each assistant and prompt pair, 10,000 times, with a fixed seed. - **Leave-one-out:** each headline finding is recomputed with each prompt removed and with each assistant removed. If it only holds because of one prompt or one assistant, I say so. - **Assistant vs wording** uses a permutation test, because raw gaps between small groups are inflated by noise. I label each finding by how much weight it can carry: - **Strong:** the interval excludes no effect, and the finding holds when any single prompt or assistant is dropped. - **Moderate:** the numbers are clear, but the cause is uncertain or the finding rests on my own reading of the answers. - **Tentative:** worth noting, but driven mostly by one assistant or a handful of runs. --- ## Findings | # | Finding | Numbers | Strength | |---|---|---|---| | 1 | Deel is named often and almost never recommended near the top. | Named in 41/60 (68%, CI 56 to 79%). Top three in 1/60 (2%, CI 0.3 to 9%). | Strong | | 2 | Rippling is the default first pick on both assistants. | Ranked #1 in 52/60 (87%, CI 76 to 93%). Top three in 60/60. | Strong | | 3 | The top three is a closed shop. | Rippling, Gusto, BambooHR and Justworks hold 98.9% of top-three slots. In 58/60 answers the whole top three comes from these four. | Strong | | 4 | A mention can be a warning. | Workday named in 13/60 (22%), never listed. I read all 13 and each one advised against it for a company this size. ADP and UKG were also named (7/60 each) and never listed. | Moderate | | 5 | HiBob's visibility depends on the assistant. | Named in 20/30 ChatGPT answers vs 2/30 Gemini answers. Gap of 60 points (CI 47 to 73). | Moderate | | 6 | ChatGPT names more vendors per answer. | 6.1 vs 4.8 vendors named (difference 1.3, CI 0.9 to 1.7). | Moderate | | 7 | Describing the problem changes the #1 pick. | BambooHR ranked #1 in 5/20 answers to the problem-led prompt and 0/40 to the others. 4 of the 5 were ChatGPT. | Tentative | Two notes on reading these: - **Deel's gap is about positioning, not dislike.** Both assistants treat Deel as the pick "if you hire internationally". The buyer in these prompts is a US company, so Deel gets a conditional mention and drops out of the main list. - **Findings 5 and 6 compare the assistants as I configured them.** ChatGPT ran with Thinking on High and searched the web every time. Gemini ran with extended thinking off and searched in 25 of 30 runs. Some of the difference may come from those settings, not the models themselves. ### What I did not find - **Assistant vs wording.** I set out to ask which moves a brand's mention rate more: which assistant the buyer uses, or how they phrase the question. Both have an effect beyond noise (average excess gap of 0.073 for assistant, 0.052 for wording), but neither clearly wins, and the assistant effect is mostly HiBob. I'm not drawing a conclusion here. - **Deel by assistant.** Deel was named in 77% of ChatGPT answers and 60% of Gemini answers. The interval on that gap runs from minus 3 to plus 37 points, so it is not a reliable difference. --- ## Limitations - **One category, one buyer, three prompts.** These results describe HR software for a 50-person US SaaS startup. Other categories, company sizes and phrasings may behave differently. - **Small samples.** 10 runs per prompt per assistant. The intervals are wide, and small differences between groups are noise. - **A snapshot.** Collected in October 2026. AI assistants change their models, search behavior and answers constantly. Expect these numbers to drift. - **Location.** Runs were made from outside the US without a VPN. Location can affect which sources an assistant searches and cites. - **Different settings per assistant.** See the note under Findings. - **One coder.** I coded the recommendation lists and read the Workday mentions myself. The raw answers are included so anyone can check my coding. - **Fixed name list.** Mentions are detected from a set list of vendors. A vendor not on that list would be missed. The list is in `analysis.py`. - **Two assistants.** Claude, Perplexity, Copilot and others are not in this version. --- ## Reproducing the results You need Python 3.9 or later with pandas and numpy. ``` pip install pandas numpy python analysis.py ``` The script reads `runs.csv`, rewrites `mentions.csv` and `summary.csv`, and prints every finding. It takes about two to three minutes because of the resampling. The seed is fixed, so you get the same numbers every time. --- ## How to cite ``` Jaiprakash, T. D. (2026). Mentioned Isn't Recommended: How ChatGPT and Gemini Recommend HR Software (CobaltKite Research 001) (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.23361807 ``` --- ## Version history | Version | Date | Changes | |---|---|---| | 1.0.0 | 2026-10-14 | First release. ChatGPT and Gemini, 60 runs. | I plan to rerun the same protocol every quarter and publish each round as a new version of this record, so changes over time can be tracked. Version 2.0 will add Claude. --- ## License The data and documentation are released under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). You can use, share and adapt them for any purpose, including commercial use, as long as you credit the source. --- ## Contact Tejas D Jaiprakash, Founder, CobaltKite tejas@cobaltkitecreative.com · https://cobaltkitecreative.com If you find an error in the data or the coding, please email me. Corrections will go into the next version with a note in the version history.