AI
Curated AI Jobs
Firecrawl

Research Engineer - Benchmarks

Firecrawl29 Sep 2026
fulltimeremote$250,000–$290,000 USD per year
Apply for this position

Before you apply

Listed location: San Francisco HQ; Toronto Hub

Work arrangement: remote. A remote label does not confirm worldwide eligibility or visa sponsorship.

Stated pay: $250,000–$290,000 USD per year, as published by the employer.

Read the employer’s description for qualifications, compensation and work eligibility. Confirm the position is still open on the application page.

Job description supplied by Firecrawl; category and skill labels may be inferred. How our listings work · Report a problem

Job Description

Research Engineer - Benchmarks

Every week, someone asks which data provider is actually best: for company records, for finding the right person, for fresh job listings. Right now the answers come from the vendors themselves. You'll build the independent version. You'll run rigorous, automated, public benchmarks of the providers in Alexandria and beyond, and publish them weekly. The same results will feed straight back into Alexandria so it learns which provider to call for which job.

This isn't our internal evals role. You're measuring the market, in public, where every number will be challenged by the vendors it ranks. You'll own it end to end: the datasets, the ground truth, the scoring, the harness, the weekly release, and the loop into Alexandria's provider selection. No one hands you a methodology. You write it, defend it, and ship it every week.

Salary Range: $250,000–$290,000 USD/year (SF) / $210,000–$224,000 CAD/year (Toronto)

Equity Range: Competitive equity. Details shared during the process.

Location: San Francisco, CA (SF HQ) or Toronto, ON (Toronto Hub). Hybrid, onsite 3+ days a week.

Equity Range: Competitive equity. Details shared during the process.

Location: San Francisco, CA (SF HQ). On-site, five days a week.

Job Type: Full-Time

Experience: 4+ years in ML, research engineering, or data engineering, with evaluation or benchmark work you've shipped

Work Authorization: Open to candidates already authorized to work in the US or Canada. In the US, we also support H-1B transfers, F-1 OPT and STEM OPT, TN (Canada and Mexico), and E-3 (Australia). We’re not able to sponsor new H-1B visas for candidates outside the US. For Canada, we’ll consider work permit sponsorship case by case through our Toronto Hub.

About Firecrawl

Firecrawl helps AI agents get the data they need through Alexandria (its library of official data providers), its own indexes, and the live web. Developers first turned to us because getting clean data from the web is harder than it looks. Today, Firecrawl has 200k+ stars on GitHub, and more than 1.5 million people build with it, including teams at 67 of the Fortune 100.

The open web is only part of what agents need. Much of the most useful knowledge, like company data, financial records and research, sits behind paywalls and private APIs. Firecrawl Alexandria brings that into the same API, and we pay the providers whose data agents use. We believe people who share what they know should be compensated.

Our Values

  • Bias to action: We move fast, ship hard, and don’t wait to be told.

  • Realistic about AI: We’re clear-eyed about what AI can and can’t do, and we keep pushing to use it where it actually works.

  • All in: The next few years matter a lot, and we’re all in for them.

  • Human connection: As intelligence gets cheaper, relationships with customers, partners, and each other matter more.

What You'll Do

  • Design and run head-to-head benchmarks of data providers against verified ground truth. For example:

    • Apollo vs. FullEnrich vs. DataLegion on company details.

    • FullEnrich vs. DataLegion on finding the right person and their current employer and title.

    • Built In vs. ZipRecruiter on relevant, fresh, non-duplicate job listings.

  • Build and maintain the test datasets and ground truth, and keep them from going stale or leaking.

  • Own the automated harness and the weekly benchmark release. Every run has to be reproducible, versioned, and defensible.

  • Measure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.

  • Work with marketing to ship public leaderboard pages that are useful and hold up under scrutiny.

  • Close the loop into Alexandria so benchmark results change which provider gets called for what.

  • Keep expanding the categories we benchmark, inside Alexandria and beyond it.

What We're Looking For

  • You've shipped evals or benchmarks, and you can explain exactly why your results were trustworthy.

  • You've done this somewhere that matters: a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project. OSS contributors very welcome.

  • You're strong in Python and API integrations, and comfortable with messy vendor APIs, rate limits, and inconsistent schemas.

  • You know how to build test sets and scoring methods: sampling, labeling, inter-rater agreement, and when to trust an LLM judge and when not to.

  • You use AI heavily and keep upgrading your own workflow. You ship without waiting for instructions.

  • You write findings clearly for engineers, marketers, and the vendors on the other side of the leaderboard.

What We're Not Looking For

  • Someone who wants to run benchmarks someone else designed.

  • Someone who picks the metric that makes the story look good. Our numbers have to survive the vendors who lose.

  • A pure researcher who won't build the harness, or a pure engineer who won't think hard about methodology.

  • Someone who needs a fully specced ticket, or a quarter, to ship the first result.

A Note On Pace

We operate at an absurd level of urgency because the window for what we're building won't stay open forever. If that excites you, keep reading. If it doesn't, no hard feelings, but this role probably isn't for you.

Benefits & Perks

 

Available to all employees

  • Salary that makes sense: $250,000–$290,000 USD/year, / $210,000–$224,000 CAD/year (Toronto), based on impact, not tenure

  • Own a piece: Gain competitive equity in what you're helping build

  • Generous PTO: 15 days mandatory, anything after 24 days, just ask (holidays excluded). Take the time you need to recharge

  • Parental leave: 12 weeks fully paid, for all parents

  • Wellness stipend: $100 USD/month for the gym, therapy, massages, or whatever keeps you human

  • Learning & Development: Expense up to $1,000 USD/year toward anything that helps you grow professionally

  • Team offsites: A change of scenery, minus the trust falls

  • Sabbatical: 3 paid months off after 4 years, do something fun and new

Available to US-based full-time employees

  • Full coverage, no red tape: Medical, dental, and vision (100% for employees, 50% for partner and kids). No weird loopholes, just care that works

  • Life & Disability insurance: Employer-paid basic life and AD&D, short-term disability, and long-term disability. Coverage for life's curveballs

  • Virtual care and a health guide: Teladoc for the couch doctor visit, plus Rightway to answer coverage questions and fight billing errors for you

  • Mental health: Talkspace, therapy and psychiatry on your schedule

  • Fertility and family building: Carrot, covering you and your partner

  • EAP: Free confidential counseling, legal and financial consults, and online will prep through Guardian

  • 401(k) plan: Retirement might be a ways off, but future-you will thank you

  • Pre-tax benefits: HSA, FSA, and commuter benefits to help your wallet out a bit

  • Supplemental options: Extra life and AD&D, accident, critical illness, hospital indemnity, plus pet, legal, and identity protection through MetLife

Available to Canada-based full-time employees

  • Full coverage, no red tape: Extended health, dental, and vision through Manulife (Diamond, the top tier), 100% employer-paid for you, your partner, and your kids

  • Life & Disability insurance: Employer-paid life, AD&D, short-term disability, and long-term disability. Coverage for life's curveballs

  • Virtual care: Dialogue Premium, so you can see a doctor or nurse from your couch, any hour

  • Mental health: Talkspace Elite, therapy and psychiatry on your schedule

  • Fertility and family building: Carrot, covering you and your partner

  • Retirement: Group RRSP through Wealthsimple, so future-you can thank you

Available to SF-based employees

  • SF HQ perks: Snacks, drinks, team lunches, intense ping pong, and peak startup energy

  • E-Bike transportation: A loaner electric bike to get you around the city, on us

 

Available to Toronto-based employees

  • Toronto Hub perks: Snacks, drinks, team lunches, glass-walled views down University Avenue, and a home base steps from Union Station

  • Transit, covered: A PRESTO card loaded for GO Transit, subway, and streetcar, plus station parking if you drive to the train. Winter-proof, on us

Interview Process

  1. Application Review: Send us your work. We want the benchmark, leaderboard, eval, or dataset you built, and how you knew it was right. We care about what you've shipped, not where you went to school.

  2. Intro Chat (~25 min): A quick conversation to get to know each other. We'll cover what you've been working on, what drew you to Firecrawl, and what you want next. Time for your questions too.

  3. Technical Chat (~45 min): A real problem from our world: design a benchmark that decides whether FullEnrich or DataLegion finds the right person's current title. We'll cover where the ground truth comes from and how you'd defend the result to the vendor who loses. Come ready to think out loud.

  4. Workflow Chat (~30 min): Show us how you actually work: your AI tools, your setup, and a recent thing you shipped faster than you would have a year ago.

  5. Founder Chat (~25 min): Culture, pace, ownership, and how you like to work. Time for your questions too.

  6. Paid Work Trial (~40 hours): Ship a small benchmark end to end on a real provider category, paid at a contractor rate. It's the truest signal for both sides. Remote-friendly, and we'll flex around your current commitments.

  7. Decision: We move fast after the trial.

If you want to be the person the whole market checks before picking a data provider, you should join us.

👉 Apply now.

Skills mentioned

PythonGoLLM

Categories

Frequently asked questions

Is the Research Engineer - Benchmarks position at Firecrawl remote?

Yes. The Research Engineer - Benchmarks role at Firecrawl is a remote position. Country eligibility is not specified here; check the employer listing.

What type of employment is the Research Engineer - Benchmarks role?

Firecrawl is hiring for a full-time Research Engineer - Benchmarks position.

Which skills are mentioned for the Research Engineer - Benchmarks job at Firecrawl?

Detected skill labels include Python, Go, LLM. Check the employer description to distinguish required skills from preferred experience.

How do I apply for the Research Engineer - Benchmarks position at Firecrawl?

You can apply for the Research Engineer - Benchmarks role directly through Firecrawl's official application link provided on this page.

Interested in this role?

Apply directly on the company's website.

Apply Now
Firecrawl

Firecrawl

Website
Posted29 Sep 2026
Typefulltime
LevelMid-level
LocationRemote
Apply NowView All Jobs at Firecrawl

Share this job