Culture Code: a case study

The tool itself is an invite-only beta: culture-code-iota.vercel.app.

Culture Code is a tool I built to help people vet an employer before they take a job there. You give it a company you’re considering and your own criteria. It researches the leadership and culture, then shows what it found, where it found it, and how sure it is. It never gives the company a score. I built it for my own job search, and on November 4 I opened it as an invite-only beta for other job-seekers.

A few choices shaped it. It shows how much it knows about a company instead of rating the company, the rules that matter most are enforced in code instead of left to the AI model’s good behavior, and testing on real companies turned up problems that every automated check had missed.

It also works better for some companies than others. It does best on larger companies with a lot of material online, and it runs out of things to say quickly for small ones.

The problem

Employers use a careful, multi-stage process to decide whether to hire someone. Candidates have nothing like it for deciding whether to work for an employer. Glassdoor and Blind were supposed to fill that gap, but for small companies they have little to say, they are easy to game, and they boil everything down to a star average. A star average hides the thing that tells you what a job is really like, such as how one particular leader handles disagreement.

I focused on startups and product roles. Startups are where public information about culture is thinnest, and they are small enough that one leader shapes the experience for everyone.

That choice carries a tension, which I only saw once I ran the tool on real companies. The thinner a company’s footprint, the less the tool has to work with, so it does best where the problem is smallest. When it has little to go on, I decided it should say so plainly and not fill the gap with a confident number.

What I built

You start with a company you’re already considering. The tool doesn’t find or recommend jobs. Each report follows the same seven steps:

1. Confirm the company. Names collide, so the tool first reads the company’s own homepage and asks you to confirm it has the right one. You can also add the job description and the hiring manager.
2. Research what’s public. It looks for answers to a fixed set of questions about leadership, the hiring manager, product, the team, customers, funding and working style. Every answer comes with sources, or is marked as unknown.
3. Read the job description. It looks at the language and quotes the lines it relies on, which must appear word for word in the posting. This is one source describing what the company says it wants, so confidence is capped at medium.
4. Assess. It lists red flags, checks the company against your own criteria, and compares what the company says about itself with what the evidence shows.
5. Show the gaps. It lists what public sources couldn’t answer.
6. Prepare questions for people who have left. Former employees are the most trustworthy source and can’t be automated, so the tool writes questions you can ask them.
7. Add your own notes. After you talk to people, you type in what you learned. These stay separate from the researched findings and are never treated as equal to them.

It’s a web app built with Next.js. Claude does the reading and judging, a web research service does the searching, and a full report costs roughly $0.25 to $0.35.

Design principles

The tool never scores a company. A single fit score repeats the problem with star ratings, because it hides where the uncertainty is. To see what I was avoiding, I gave the same job to a one-prompt tool in Figma Make. It produced a polished profile with a 4.4 out of 5 rating, 87% positive sentiment and values scored like 94, and it cited no sources and admitted no unknowns. I borrowed a few ideas from it, like how it described working style, and rejected every number, because a number implies more certainty than public information can support.

Instead, each claim gets its own confidence level (high, medium or low) and its own sources. The only meter in the app says how much we know, which describes the evidence and not the company. Confidence also depends on who is speaking. If the only support for a claim is the company, its founders or its employees, the claim is rated low and labeled the company’s own account. High confidence needs two independent outlets.

Ratings from review sites are allowed only with their source and how many people gave them, such as a 2.3 on the App Store from 572 ratings. A bare 2.8 out of 5 is exactly the Glassdoor problem, so the code removes it, along with approval scores for named leaders and any rating in a headline.

When the tool can’t find something, it says so. Every question ends up answered or marked as having no public information. If a search failed, it’s marked not checked, which is different from nothing being out there.

The rules that matter most are enforced in code. An AI model follows instructions most of the time, and in a report about real companies the occasional lapse is the worst failure I could have. So every claim needs a source that has been verified, only founders, executives and the hiring manager can be named, and quotes from a job description must appear in it word for word. None of this relies on the model getting it right.

Key decisions and pivots

Decision	Why
Open it as an invite-only beta for other job-seekers	I wanted to learn whether anyone else wants it without risking runaway costs. Each person’s data is kept separate, research isn’t shared between users, usage is capped, and only invited people can sign in.
Allow ratings, but only with their source and count	My second round of testing showed that banning ratings outright let some through in phrasings the filter missed, and deleted too much when it did catch them. A rating with its sample size is context a reader can weigh.
Cut resume matching and application tracking, and assess culture only	The tool had drifted into an application tracker. Comparing me to a job description says nothing about whether the culture is good.
Drop the application tracker idea	Notion already handles tracking, and the extra features answered a different question from “can I trust this employer?”

How I tested it

The most useful thing testing taught me is that every automated check can pass while someone who knows the company still finds the report wrong. I ran the tool on seven real startups in two rounds. Code checked structure and sourcing, and I judged everything else by hand.

Test	What it measured	Result
First round, automated checks	Sources on every finding, correct company, quotes matching the posting	All passed
First round, accuracy	Were the red flags and contradictions real?	All 13 red flags and all 6 contradictions were real
First round, confidence	Did the report’s confidence match my own judgment?	I agreed with 64% of ratings, against a bar of 80%
Second round, automated checks	The same checks after fixes	All passed
Same-name source filter	Correct evidence removed by mistake	None across 3 runs, after a redesign
AI checker, while tuning it	Agreement with my own labels	20%, then 56%, then 68%, then 84% after I corrected four of my labels
AI checker, second round	Claims fully supported by their cited sources	43 of 70 (61%), against a target of 90%

In the first round the research was accurate, but the confidence ratings were not. I agreed with only 64% of them, and every disagreement went the same way: the report sounded too sure of claims that came from the company talking about itself. It treated a CEO’s single LinkedIn post about the product as strong evidence, took a CEO’s description of the culture as calm and humble as proof that it was, and read a help center’s release notes as proof that the product ships. Only someone who knew the company could have caught this, and all of the automated checks passed anyway. So I made confidence depend on who is speaking, and I did it in code so it also covers reports already saved. Hard facts like a funding round or a CEO change are exempt, because a press release is a reliable source for those.

The tool also had to deal with sources about the wrong company, since names collide. I added a step that reviews every source and drops the ones that belong to a different company with the same name. My first attempt ran alongside other work and caught different sources on different runs. When I made it a dedicated step it got worse and started removing correct evidence, a different piece each time. It removed a lawsuit the company was a party to, because the suit was filed under the company’s legal name, and a review that mentioned what the company used to be called. It settled down once I required a verdict and a reason for every source, gave it the identity facts the research had already verified, and exempted the company’s own website in code. Across three runs after that it made no wrong removals, and for a company that shares its name with an energy utility in another country, it removed only the utility’s pages. I took two things from this. A filter that is asked to list the bad items makes quiet, random mistakes, while one that has to judge every item is steady. And a filter can delete good evidence without anyone noticing, so it needs its own test for that, in addition to a test that it catches the bad sources.

My second round of testing found problems with the rule against showing ratings. Some phrasings got through, like “rated leadership a C-” and “2.3 from 572 ratings”. And when the filter did catch a rating, it deleted the whole sentence, which once left a red flag with a headline and no explanation. So I changed the rule to allow ratings with their source and count, and changed the code to remove only the offending phrase and to drop any flag left without context.

To check whether each claim matches what its cited sources actually say, I built a second AI checker and compared it to my own judgments. At first we agreed only 20% of the time. Putting contradictions first in its instructions raised that to 56%, but it still described a reversed customer relationship as partly supported. It improved when I had it list three things separately (facts the source contradicts, central facts the source leaves out, and minor details it leaves out) and let code decide the verdict from the lists. Agreement went to 68%, and it caught every error I had planted. I then went back over the six claims where it was stricter than I had been. On four of them it was right: no source supported the claim, and I had passed it. After I corrected those labels, we agreed 84% of the time.

Running it on 70 claims from the second round, I found that only 61% were fully supported by their sources. My earlier hand checks of five claims at a time had suggested 80 to 100%, so the small samples had flattered the product. The larger run turned up wrong years and posts credited to the wrong person. I now use the checker as a first pass and read only the claims it fails. Code makes the final call, so the checker can’t name a contradiction and still pass a claim.

The seven companies also showed the limit I mentioned at the start: the tool works better for larger companies with more online presence. The company with the smallest footprint got answers to 10 of its 14 questions and one other got 11, where most got 12, and three of the seven had no employee reviews it could read. When there’s little to go on, the report says so, which is what I wanted. It also means the tool has the least to tell you about smaller companies.

Two failures worth knowing about

The first is companies with similar names. The research searches by name, and names collide. One of my test companies shares a name with an AI company, and another with an energy utility in another country. Reading the company’s homepage identified the right company on the first try, but a blog post from the similarly named AI company still got into a finding during the full research. Telling the model not to use it didn’t stop that. A code rule did: drop any source from a different website that carries the company’s name. Identity is now checked at several points: the homepage, your confirmation, citing only results the tool verified, that website rule, and the source review described above.

The second came from an answer that looked complete but wasn’t. Early on, one call to the AI used 1,458 of the 1,500 tokens I had allowed for its answer and silently cut off its last field. The answer still looked valid and no error was raised. Now every call gets generous room, checks that it finished and that every required field is present, and fails loudly instead of saving an incomplete result.

What’s next

The biggest open problem is getting insider input. The best source is people who worked at the company, and right now that part is entirely manual, with no reason for anyone to contribute. It’s the same gap that weakens Glassdoor for small companies, and it matters most for exactly the companies where this tool finds the least online. I’ve been thinking about reaching out to people who were recently laid off. They know the company, and they’re looking for work themselves. In exchange for notes, I could give them free research runs, which cost me about $0.25 to $0.35 each. There are real risks. Laid-off people can be bitter or relieved, so the sample skews, and notes about named leaders raise consent and legal questions. Whatever I do, those notes stay labeled as personal accounts and never count as equal to the researched findings. Before doing it at scale I’d try it with about ten people from one or two companies.

The tool is also built around product managers right now, but only two of its 14 research questions are specific to them: whether product has real ownership, and whether the PM seat has turned over. The rest apply to any role. I’d let people pick their role and swap in a role-specific topic, starting with product, engineering and design, and test the new questions the same way before releasing them.

The design also needs to look more professional before I put it in front of strangers. Smaller items are still on the list: including insider notes in a refreshed report, suggesting the hiring manager from the “reports to” line in a job posting, measuring how often the AI checker is right when it fails a claim, and tightening how dates and attributions are handled in the research.

If I did it again, I’d run the AI checker from the start, since my five-claim hand checks made the product look better than it was.