diff --git a/adr/0001-project-scoping-for-server-functions.md b/specs/0001-project-scoping-for-server-functions.md similarity index 100% rename from adr/0001-project-scoping-for-server-functions.md rename to specs/0001-project-scoping-for-server-functions.md diff --git a/adr/0002-hosted-dataforseo-metering-with-autumn.md b/specs/0002-hosted-dataforseo-metering-with-autumn.md similarity index 100% rename from adr/0002-hosted-dataforseo-metering-with-autumn.md rename to specs/0002-hosted-dataforseo-metering-with-autumn.md diff --git a/adr/0003-google-search-console-integration.md b/specs/0003-google-search-console-integration.md similarity index 100% rename from adr/0003-google-search-console-integration.md rename to specs/0003-google-search-console-integration.md diff --git a/specs/0004-keyword-data-source-routing.md b/specs/0004-keyword-data-source-routing.md new file mode 100644 index 0000000..16cde43 --- /dev/null +++ b/specs/0004-keyword-data-source-routing.md @@ -0,0 +1,110 @@ +# Keyword data source routing and the clickstream default + +## Status + +Accepted (June 2026) + +## Context + +Keyword research is the largest credit spend for OpenSEO users, and coverage +has a hard gap: DataForSEO Labs supports 94 countries, so a customer in +Iceland (location 2352) cannot run keyword research at all. + +Three data sources were on the table: + +- **DataForSEO Labs** — our existing source. Per-row pricing ($0.01/task + + $0.0001/row), and the only source for keyword difficulty, search intent, + and SERP-feature context. Its `include_clickstream_data` flag doubles the + request cost; its only effect is refined volume numbers — the standard + `keyword_info.search_volume` is the same Google-Ads-derived volume every + mainstream tool shows. +- **DataForSEO Keywords Data (Google Ads endpoints)** — same vendor, flat + $0.075 per live request (up to 1,000 keywords for `search_volume`, up to + 20 seeds for `keywords_for_keywords`), 217 countries including Iceland. No + difficulty, intent, or SERP data; volumes are bucketed and aggregate close + variants. +- **Direct Google Ads API** — free, but not usable in a SaaS: Google's + Targeting-data policy forbids collecting Keyword Planner data "for any + purposes other than creating or managing Google Ads campaigns," and + exposing it to users requires the full Required Minimum Functionality (a + campaign-management suite). Volumes are bucketed without active ad spend. + +## Decision + +Every supported country has exactly one keyword-data provider, resolved by +`getKeywordDataProvider(locationCode)` in `src/shared/keyword-locations.ts`. +There is no user-facing provider choice. + +1. **Labs is the default provider.** Countries Labs does not cover are + flagged `googleAdsOnly` in `LOCATION_OPTIONS` and are served by the + Keywords Data Google Ads endpoints. Unknown location codes fall back to + Labs, which rejects them with its own error. +2. **Routing per feature:** + + | Feature | Labs country | Google-Ads-only country (e.g. Iceland) | + | --------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------- | + | Keyword research (UI + `research_keywords`) | Labs related→suggestions→ideas | `keywords_for_keywords` (single source) | + | `get_keyword_metrics`, rank-tracking metric refresh | Labs keyword_overview | `search_volume` | + | SERP analysis, `get_serp_results`, rank tracking | SERP API | SERP API (supports all countries) | + | Domain overview, ranked keywords, SERP competitors | Labs | **Unavailable** — pickers filtered to Labs countries; MCP tools return a clear validation error | + +3. **Google-Ads-sourced rows carry no keyword difficulty or intent** + (`keywordDifficulty: null`, `intent: "unknown"`). The research page and + the MCP tool descriptions state this whenever such a country is in play. +4. **Clickstream refinement is opt-in per call and defaults to off**, for + the Labs research endpoints (related/suggestions/ideas) and keyword + overview. Users opt in via the labeled checkbox on the research page + (URL param `cs`, carried per keyword tab, hidden for Google-Ads-only + countries) or via `includeClickstreamData` on the `research_keywords` and + `get_keyword_metrics` MCP tools. The label and tool descriptions must + state the 2× credit cost. The flag is part of the research cache key. +5. **Language codes for Google-Ads-only countries must exist in both the + Google Ads and SERP language lists** — the country picker is shared with + rank tracking, which uses the SERP API. China is excluded: its Ads + language code (`zh_CN`) conflicts with the SERP format (`zh-CN`), and + Google search does not meaningfully operate there. +6. **Billing is unchanged.** `keywords_data/*` task costs flow through the + same envelope → markup → Autumn pipeline as Labs calls and map to the + `keyword_research` credit feature (rank tracking overrides to + `rank_tracking`). + +## Rationale + +Cost at our actual defaults (research default = 150 rows/seed; credits = +USD × 1.28 markup × 1000): + +| Call | Labs (with clickstream) | Labs (default) | Google Ads | +| ------------------ | ----------------------- | -------------- | -------------- | +| research, 150 rows | $0.050 → 64 cr | $0.025 → 32 cr | $0.075 → 96 cr | +| research, 500 rows | $0.120 → 154 cr | $0.060 → 77 cr | $0.075 → 96 cr | +| metrics, 100 kw | $0.020 → 26 cr | same | $0.075 → 96 cr | +| metrics, 700 kw | $0.080 → 103 cr | same | $0.075 → 96 cr | + +- A wholesale switch to Google Ads data would raise the cost of typical + calls and lose difficulty/intent; replicating difficulty alone via + `bulk_keyword_difficulty` ($0.11/1k) erases any savings. Hybrid keeps the + better data where it exists and adds coverage where it doesn't. +- Always-on clickstream silently doubled the #1 spend feature for a marginal + volume refinement. Off-by-default halves default research cost + (~64 → ~32 credits per seed); the opt-in keeps the refinement available to + users who want it, priced visibly. +- The direct Google Ads API is rejected on policy, not effort — revisit only + if OpenSEO ships campaign management. + +## Consequences + +- Iceland and ~47 other countries are selectable for keyword research and + rank tracking; a Google-Ads-served research or metrics call costs a flat + ~96 credits. +- Cross-country volume numbers stay roughly comparable: both providers' + standard volumes derive from Google Ads data. +- Google Ads live endpoints allow 12 requests/min per DataForSEO account. + Research fans out at most 5 seeds per call, so a single user stays under + it; sustained multi-user traffic on these countries would queue. +- `keywords_for_keywords` has no limit parameter (up to 20k suggestions per + flat-fee request); results are sorted by volume server-side and truncated + to the requested limit. +- The research cache version was bumped (2→3) so pre-change + clickstream-priced volumes never mix with standard ones. +- Reverting the clickstream default is a one-line change per fetcher in + `src/server/lib/dataforseo/labs.ts`; the opt-in plumbing stays either way. diff --git a/src/client/features/domain/DomainOverviewPage.tsx b/src/client/features/domain/DomainOverviewPage.tsx index c812ebc..558e7df 100644 --- a/src/client/features/domain/DomainOverviewPage.tsx +++ b/src/client/features/domain/DomainOverviewPage.tsx @@ -11,7 +11,7 @@ import { DEFAULT_LOCATION_CODE, LOCATIONS, getLanguageCode, - isSupportedLocationCode, + isLabsLocationCode, } from "@/client/features/keywords/locations"; import { useDomainSearchHistory } from "@/client/hooks/useDomainSearchHistory"; import type { DomainSearchHistoryItem } from "@/client/hooks/useDomainSearchHistory"; @@ -124,7 +124,7 @@ function getHistorySearchUpdate( item: DomainSearchHistoryItem, ): DomainSearchUpdate { const historyLocation = - item.locationCode != null && isSupportedLocationCode(item.locationCode) + item.locationCode != null && isLabsLocationCode(item.locationCode) ? item.locationCode : DEFAULT_LOCATION_CODE; diff --git a/src/client/features/domain/components/DomainSearchCard.tsx b/src/client/features/domain/components/DomainSearchCard.tsx index 6ffa15f..5d1011b 100644 --- a/src/client/features/domain/components/DomainSearchCard.tsx +++ b/src/client/features/domain/components/DomainSearchCard.tsx @@ -4,7 +4,7 @@ import { getFieldError, getFormError } from "@/client/lib/forms"; import type { DomainOverviewControlsForm } from "@/client/features/domain/DomainOverviewPage"; import { toSortMode } from "@/client/features/domain/utils"; import type { DomainSortMode } from "@/client/features/domain/types"; -import { LOCATION_OPTIONS } from "@/client/features/keywords/locations"; +import { LABS_LOCATION_OPTIONS } from "@/client/features/keywords/locations"; type Props = { controlsForm: DomainOverviewControlsForm; @@ -63,7 +63,7 @@ export function DomainSearchCard({ onLocationChange(next); }} > - {LOCATION_OPTIONS.map((option) => ( + {LABS_LOCATION_OPTIONS.map((option) => ( diff --git a/src/client/features/domain/domainRouteState.ts b/src/client/features/domain/domainRouteState.ts index 7e74e46..b243fbb 100644 --- a/src/client/features/domain/domainRouteState.ts +++ b/src/client/features/domain/domainRouteState.ts @@ -4,7 +4,7 @@ import { } from "@/types/schemas/domain"; import { DEFAULT_LOCATION_CODE, - isSupportedLocationCode, + isLabsLocationCode, } from "@/client/features/keywords/locations"; import { EMPTY_DOMAIN_FILTERS, @@ -46,8 +46,9 @@ export function getDomainRouteState( search: DomainSearchParams, ): DomainOverviewRouteState { const normalizedSort = toSortMode(search.sort ?? null) ?? "traffic"; + // Domain analytics is Labs-backed; Google-Ads-only countries aren't valid. const normalizedLocationCode = - search.loc != null && isSupportedLocationCode(search.loc) + search.loc != null && isLabsLocationCode(search.loc) ? search.loc : DEFAULT_LOCATION_CODE; diff --git a/src/client/features/keywords/hooks/useKeywordControlsForm.ts b/src/client/features/keywords/hooks/useKeywordControlsForm.ts index 5b1db08..b9b19b9 100644 --- a/src/client/features/keywords/hooks/useKeywordControlsForm.ts +++ b/src/client/features/keywords/hooks/useKeywordControlsForm.ts @@ -16,6 +16,7 @@ type KeywordTabValidationInput = { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; type UseKeywordControlsFormInput = { @@ -23,6 +24,7 @@ type UseKeywordControlsFormInput = { locationCode: number; resultLimit: ResultLimit; keywordMode: KeywordMode; + clickstream: boolean; getOpenKeywordTabs?: () => readonly KeywordTabValidationInput[]; keywordTabsLimit?: number; }; @@ -32,6 +34,7 @@ export type KeywordControlsValues = { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; function getKeywordSearchValidationErrors( @@ -82,6 +85,7 @@ function getKeywordTabCapacityError( locationCode: value.locationCode, resultLimit: value.resultLimit, mode: value.mode, + clickstream: value.clickstream, }; const alreadyOpen = simulatedOpenTabs.some((tab) => keywordTabMatches(tab, input), @@ -113,7 +117,8 @@ function keywordTabMatches( tab.keyword === input.keyword && tab.locationCode === input.locationCode && tab.resultLimit === input.resultLimit && - tab.mode === input.mode + tab.mode === input.mode && + tab.clickstream === input.clickstream ); } @@ -127,6 +132,7 @@ export function useKeywordControlsForm( locationCode: input.locationCode, resultLimit: input.resultLimit, mode: input.keywordMode, + clickstream: input.clickstream, }, validators: { onChange: ({ formApi, value }) => @@ -154,6 +160,7 @@ export function useKeywordControlsForm( locationCode: input.locationCode, resultLimit: input.resultLimit, mode: input.keywordMode, + clickstream: input.clickstream, }); }, [ form, @@ -161,6 +168,7 @@ export function useKeywordControlsForm( input.keywordMode, input.locationCode, input.resultLimit, + input.clickstream, ]); return form; diff --git a/src/client/features/keywords/hooks/useKeywordResearchData.ts b/src/client/features/keywords/hooks/useKeywordResearchData.ts index d4ec9a3..a2a08e7 100644 --- a/src/client/features/keywords/hooks/useKeywordResearchData.ts +++ b/src/client/features/keywords/hooks/useKeywordResearchData.ts @@ -8,7 +8,7 @@ import { parseKeywordInput } from "@/client/features/keywords/state/keywordContr import { researchKeywords } from "@/serverFunctions/keywords"; import type { KeywordMode, - KeywordSource, + ResearchSource, ResultLimit, } from "@/client/features/keywords/keywordResearchTypes"; @@ -24,6 +24,7 @@ type KeywordResearchQueryInput = { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; type KeywordResearchRequest = { @@ -34,6 +35,7 @@ type KeywordResearchRequest = { languageCode: string; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; export const KEYWORD_RESEARCH_STALE_TIME_MS = 24 * 60 * 60 * 1000; @@ -53,6 +55,7 @@ export function buildKeywordResearchRequest( languageCode: getLanguageCode(input.locationCode), resultLimit: input.resultLimit, mode: input.mode, + clickstream: input.clickstream, }; } @@ -68,6 +71,7 @@ export function buildKeywordResearchQueryKey( request.languageCode, request.resultLimit, request.mode, + request.clickstream, ] : ["keywordResearch", "idle"]; } @@ -81,6 +85,7 @@ export function keywordResearchQueryFn(request: KeywordResearchRequest) { languageCode: request.languageCode, resultLimit: request.resultLimit, mode: request.mode, + clickstream: request.clickstream, }, }); } @@ -89,7 +94,14 @@ export function useKeywordResearchData( input: KeywordResearchQueryInput, addSearch: AddSearchFn, ) { - const { keywordInput, locationCode, mode, projectId, resultLimit } = input; + const { + clickstream, + keywordInput, + locationCode, + mode, + projectId, + resultLimit, + } = input; const request = useMemo( () => buildKeywordResearchRequest({ @@ -98,8 +110,9 @@ export function useKeywordResearchData( mode, projectId, resultLimit, + clickstream, }), - [keywordInput, locationCode, mode, projectId, resultLimit], + [clickstream, keywordInput, locationCode, mode, projectId, resultLimit], ); const queryKey = useMemo( () => buildKeywordResearchQueryKey(request), @@ -133,6 +146,7 @@ export function useKeywordResearchData( captureClientEvent("keyword_research:search_complete", { location_code: request.locationCode, search_mode: request.mode, + clickstream: request.clickstream, result_count: researchQuery.data.rows.length, }); @@ -161,7 +175,7 @@ export function useKeywordResearchData( hasSearched, lastSearchError: hasSearched && researchQuery.isError, lastResultSource: - researchQuery.data?.source ?? ("related" as KeywordSource), + researchQuery.data?.source ?? ("related" as ResearchSource), lastUsedFallback: researchQuery.data?.usedFallback ?? false, lastSearchKeyword: request?.seedKeyword ?? "", lastSearchLocationCode: request?.locationCode ?? DEFAULT_LOCATION_CODE, diff --git a/src/client/features/keywords/keywordResearchTypes.ts b/src/client/features/keywords/keywordResearchTypes.ts index 37578f5..6071980 100644 --- a/src/client/features/keywords/keywordResearchTypes.ts +++ b/src/client/features/keywords/keywordResearchTypes.ts @@ -5,6 +5,8 @@ export const RESULT_LIMITS: ResultLimit[] = [150, 300, 500]; export type KeywordSource = "related" | "suggestions" | "ideas"; export type KeywordMode = "auto" | KeywordSource; +/** Actual result source; google_ads serves countries Labs doesn't cover. */ +export type ResearchSource = KeywordSource | "google_ads"; export type KeywordFilterValues = { include: string; diff --git a/src/client/features/keywords/keywordSearchParams.ts b/src/client/features/keywords/keywordSearchParams.ts index 09ff9d8..b5d1874 100644 --- a/src/client/features/keywords/keywordSearchParams.ts +++ b/src/client/features/keywords/keywordSearchParams.ts @@ -9,6 +9,7 @@ type KeywordSearchParams = { loc?: number; kLimit?: ResultLimit; mode?: KeywordMode; + cs?: boolean; sort?: SortField; order?: SortDir; minVol?: string; @@ -31,6 +32,7 @@ export function normalizeLegacyKeywordSearch(search: KeywordSearchParams): { loc: search.loc, kLimit: search.kLimit === 150 ? undefined : search.kLimit, mode: search.mode === "auto" ? undefined : search.mode, + cs: search.cs === true ? true : undefined, sort: search.sort === "searchVolume" ? undefined : search.sort, order: search.order === "desc" ? undefined : search.order, minVol: undefined, @@ -48,6 +50,7 @@ export function normalizeLegacyKeywordSearch(search: KeywordSearchParams): { "loc", "kLimit", "mode", + "cs", "sort", "order", "minVol", diff --git a/src/client/features/keywords/locations.ts b/src/client/features/keywords/locations.ts index 4229923..cdee9a9 100644 --- a/src/client/features/keywords/locations.ts +++ b/src/client/features/keywords/locations.ts @@ -1,156 +1,11 @@ -/** - * DataForSEO-supported countries. - * - * Source: https://cdn.dataforseo.com/v3/locations/locations_and_languages_dataforseo_labs_2026_04_06.csv - * - * For countries with multiple Google-supported languages, we pick the - * language with the largest keyword corpus (the primary search market) - * as the default. DataForSEO Labs APIs accept a single location_code + - * language_code pair per request, so we expose one entry per country. - * - * Entries are sorted alphabetically by country name; pick US as the - * product-wide default via DEFAULT_LOCATION_CODE below. - */ -export const DEFAULT_LOCATION_CODE = 2840; - -export const LOCATION_OPTIONS = [ - { code: 2008, label: "Albania", shortLabel: "AL", languageCode: "sq" }, - { code: 2012, label: "Algeria", shortLabel: "DZ", languageCode: "fr" }, - { code: 2024, label: "Angola", shortLabel: "AO", languageCode: "pt" }, - { code: 2032, label: "Argentina", shortLabel: "AR", languageCode: "es" }, - { code: 2051, label: "Armenia", shortLabel: "AM", languageCode: "hy" }, - { code: 2036, label: "Australia", shortLabel: "AU", languageCode: "en" }, - { code: 2040, label: "Austria", shortLabel: "AT", languageCode: "de" }, - { code: 2031, label: "Azerbaijan", shortLabel: "AZ", languageCode: "az" }, - { code: 2048, label: "Bahrain", shortLabel: "BH", languageCode: "ar" }, - { code: 2050, label: "Bangladesh", shortLabel: "BD", languageCode: "bn" }, - { code: 2056, label: "Belgium", shortLabel: "BE", languageCode: "nl" }, - { code: 2068, label: "Bolivia", shortLabel: "BO", languageCode: "es" }, - { - code: 2070, - label: "Bosnia and Herzegovina", - shortLabel: "BA", - languageCode: "bs", - }, - { code: 2076, label: "Brazil", shortLabel: "BR", languageCode: "pt" }, - { code: 2100, label: "Bulgaria", shortLabel: "BG", languageCode: "bg" }, - { code: 2854, label: "Burkina Faso", shortLabel: "BF", languageCode: "fr" }, - { code: 2116, label: "Cambodia", shortLabel: "KH", languageCode: "en" }, - { code: 2120, label: "Cameroon", shortLabel: "CM", languageCode: "fr" }, - { code: 2124, label: "Canada", shortLabel: "CA", languageCode: "en" }, - { code: 2152, label: "Chile", shortLabel: "CL", languageCode: "es" }, - { code: 2170, label: "Colombia", shortLabel: "CO", languageCode: "es" }, - { code: 2188, label: "Costa Rica", shortLabel: "CR", languageCode: "es" }, - { code: 2384, label: "Cote d'Ivoire", shortLabel: "CI", languageCode: "fr" }, - { code: 2191, label: "Croatia", shortLabel: "HR", languageCode: "hr" }, - { code: 2196, label: "Cyprus", shortLabel: "CY", languageCode: "el" }, - { code: 2203, label: "Czechia", shortLabel: "CZ", languageCode: "cs" }, - { code: 2208, label: "Denmark", shortLabel: "DK", languageCode: "da" }, - { code: 2218, label: "Ecuador", shortLabel: "EC", languageCode: "es" }, - { code: 2818, label: "Egypt", shortLabel: "EG", languageCode: "ar" }, - { code: 2222, label: "El Salvador", shortLabel: "SV", languageCode: "es" }, - { code: 2233, label: "Estonia", shortLabel: "EE", languageCode: "et" }, - { code: 2246, label: "Finland", shortLabel: "FI", languageCode: "fi" }, - { code: 2250, label: "France", shortLabel: "FR", languageCode: "fr" }, - { code: 2276, label: "Germany", shortLabel: "DE", languageCode: "de" }, - { code: 2288, label: "Ghana", shortLabel: "GH", languageCode: "en" }, - { code: 2300, label: "Greece", shortLabel: "GR", languageCode: "el" }, - { code: 2320, label: "Guatemala", shortLabel: "GT", languageCode: "es" }, - { code: 2344, label: "Hong Kong", shortLabel: "HK", languageCode: "zh-TW" }, - { code: 2348, label: "Hungary", shortLabel: "HU", languageCode: "hu" }, - { code: 2356, label: "India", shortLabel: "IN", languageCode: "en" }, - { code: 2360, label: "Indonesia", shortLabel: "ID", languageCode: "id" }, - { code: 2372, label: "Ireland", shortLabel: "IE", languageCode: "en" }, - { code: 2376, label: "Israel", shortLabel: "IL", languageCode: "he" }, - { code: 2380, label: "Italy", shortLabel: "IT", languageCode: "it" }, - { code: 2392, label: "Japan", shortLabel: "JP", languageCode: "ja" }, - { code: 2400, label: "Jordan", shortLabel: "JO", languageCode: "ar" }, - { code: 2398, label: "Kazakhstan", shortLabel: "KZ", languageCode: "ru" }, - { code: 2404, label: "Kenya", shortLabel: "KE", languageCode: "en" }, - { code: 2428, label: "Latvia", shortLabel: "LV", languageCode: "lv" }, - { code: 2440, label: "Lithuania", shortLabel: "LT", languageCode: "lt" }, - { code: 2458, label: "Malaysia", shortLabel: "MY", languageCode: "en" }, - { code: 2470, label: "Malta", shortLabel: "MT", languageCode: "en" }, - { code: 2484, label: "Mexico", shortLabel: "MX", languageCode: "es" }, - { code: 2498, label: "Moldova", shortLabel: "MD", languageCode: "ro" }, - { code: 2492, label: "Monaco", shortLabel: "MC", languageCode: "fr" }, - { code: 2504, label: "Morocco", shortLabel: "MA", languageCode: "ar" }, - { - code: 2104, - label: "Myanmar (Burma)", - shortLabel: "MM", - languageCode: "en", - }, - { code: 2528, label: "Netherlands", shortLabel: "NL", languageCode: "nl" }, - { code: 2554, label: "New Zealand", shortLabel: "NZ", languageCode: "en" }, - { code: 2558, label: "Nicaragua", shortLabel: "NI", languageCode: "es" }, - { code: 2566, label: "Nigeria", shortLabel: "NG", languageCode: "en" }, - { - code: 2807, - label: "North Macedonia", - shortLabel: "MK", - languageCode: "mk", - }, - { code: 2578, label: "Norway", shortLabel: "NO", languageCode: "nb" }, - { code: 2586, label: "Pakistan", shortLabel: "PK", languageCode: "en" }, - { code: 2591, label: "Panama", shortLabel: "PA", languageCode: "es" }, - { code: 2600, label: "Paraguay", shortLabel: "PY", languageCode: "es" }, - { code: 2604, label: "Peru", shortLabel: "PE", languageCode: "es" }, - { code: 2608, label: "Philippines", shortLabel: "PH", languageCode: "en" }, - { code: 2616, label: "Poland", shortLabel: "PL", languageCode: "pl" }, - { code: 2620, label: "Portugal", shortLabel: "PT", languageCode: "pt" }, - { code: 2642, label: "Romania", shortLabel: "RO", languageCode: "ro" }, - { code: 2682, label: "Saudi Arabia", shortLabel: "SA", languageCode: "ar" }, - { code: 2686, label: "Senegal", shortLabel: "SN", languageCode: "fr" }, - { code: 2688, label: "Serbia", shortLabel: "RS", languageCode: "sr" }, - { code: 2702, label: "Singapore", shortLabel: "SG", languageCode: "en" }, - { code: 2703, label: "Slovakia", shortLabel: "SK", languageCode: "sk" }, - { code: 2705, label: "Slovenia", shortLabel: "SI", languageCode: "sl" }, - { code: 2710, label: "South Africa", shortLabel: "ZA", languageCode: "en" }, - { code: 2410, label: "South Korea", shortLabel: "KR", languageCode: "ko" }, - { code: 2724, label: "Spain", shortLabel: "ES", languageCode: "es" }, - { code: 2144, label: "Sri Lanka", shortLabel: "LK", languageCode: "en" }, - { code: 2752, label: "Sweden", shortLabel: "SE", languageCode: "sv" }, - { code: 2756, label: "Switzerland", shortLabel: "CH", languageCode: "de" }, - { code: 2158, label: "Taiwan", shortLabel: "TW", languageCode: "zh-TW" }, - { code: 2764, label: "Thailand", shortLabel: "TH", languageCode: "th" }, - { code: 2788, label: "Tunisia", shortLabel: "TN", languageCode: "ar" }, - { code: 2792, label: "Turkiye", shortLabel: "TR", languageCode: "tr" }, - { code: 2804, label: "Ukraine", shortLabel: "UA", languageCode: "uk" }, - { - code: 2784, - label: "United Arab Emirates", - shortLabel: "AE", - languageCode: "en", - }, - { - code: 2826, - label: "United Kingdom", - shortLabel: "UK", - languageCode: "en", - }, - { code: 2840, label: "United States", shortLabel: "US", languageCode: "en" }, - { code: 2858, label: "Uruguay", shortLabel: "UY", languageCode: "es" }, - { code: 2862, label: "Venezuela", shortLabel: "VE", languageCode: "es" }, - { code: 2704, label: "Vietnam", shortLabel: "VN", languageCode: "vi" }, -] as const; - -const LOCATION_CODES = new Set( - LOCATION_OPTIONS.map((option) => option.code), -); - -export const LOCATIONS: Record = Object.fromEntries( - LOCATION_OPTIONS.map((option) => [option.code, option.shortLabel]), -); - -const LOCATION_LANGUAGE: Record = Object.fromEntries( - LOCATION_OPTIONS.map((option) => [option.code, option.languageCode]), -); - -export function getLanguageCode(locationCode: number): string { - return LOCATION_LANGUAGE[locationCode] ?? "en"; -} - -export function isSupportedLocationCode(locationCode: number): boolean { - return LOCATION_CODES.has(locationCode); -} +// Location data lives in shared/ so the server can route keyword research by +// provider (Labs vs Google Ads). This shim keeps existing client imports. +export { + DEFAULT_LOCATION_CODE, + LABS_LOCATION_OPTIONS, + LOCATION_OPTIONS, + LOCATIONS, + getLanguageCode, + isLabsLocationCode, + isSupportedLocationCode, +} from "@/shared/keyword-locations"; diff --git a/src/client/features/keywords/page/KeywordResearchPage.tsx b/src/client/features/keywords/page/KeywordResearchPage.tsx index 5f838c8..8442fb5 100644 --- a/src/client/features/keywords/page/KeywordResearchPage.tsx +++ b/src/client/features/keywords/page/KeywordResearchPage.tsx @@ -40,6 +40,7 @@ export function KeywordResearchPage(input: Props) { loc: undefined, kLimit: undefined, mode: undefined, + cs: undefined, }); return; } @@ -52,6 +53,7 @@ export function KeywordResearchPage(input: Props) { : tabInput.locationCode, kLimit: tabInput.resultLimit === 150 ? undefined : tabInput.resultLimit, mode: tabInput.mode === "auto" ? undefined : tabInput.mode, + cs: tabInput.clickstream ? true : undefined, }); }, [setSearchParams], @@ -67,8 +69,10 @@ export function KeywordResearchPage(input: Props) { locationCode: input.locationCode, resultLimit: input.resultLimit, mode: input.keywordMode, + clickstream: input.clickstream, }; }, [ + input.clickstream, input.keywordInput, input.keywordMode, input.locationCode, @@ -108,6 +112,7 @@ export function KeywordResearchPage(input: Props) { locationCode: value.locationCode, resultLimit: value.resultLimit, mode: value.mode, + clickstream: value.clickstream, })); let activeInput: KeywordSearchTabInput | null = null; @@ -135,6 +140,7 @@ export function KeywordResearchPage(input: Props) { locationCode: tab.input.locationCode, resultLimit: tab.input.resultLimit, mode: tab.input.mode, + clickstream: tab.input.clickstream, }, ] : [], @@ -152,6 +158,7 @@ export function KeywordResearchPage(input: Props) { hasExplicitLocationCode: true, resultLimit: activeTab.input.resultLimit, keywordMode: activeTab.input.mode, + clickstream: activeTab.input.clickstream, getOpenKeywordTabs, keywordTabsLimit: searchTabs.limit, } diff --git a/src/client/features/keywords/page/KeywordResearchSearchBar.tsx b/src/client/features/keywords/page/KeywordResearchSearchBar.tsx index a7ae130..c8d9d5e 100644 --- a/src/client/features/keywords/page/KeywordResearchSearchBar.tsx +++ b/src/client/features/keywords/page/KeywordResearchSearchBar.tsx @@ -1,4 +1,4 @@ -import { Search } from "lucide-react"; +import { Info, Search } from "lucide-react"; import { getFieldError } from "@/client/lib/forms"; import { isResultLimit, @@ -8,7 +8,10 @@ import { MAX_KEYWORDS_PER_SUBMIT, RESULT_LIMITS, } from "@/client/features/keywords/keywordResearchTypes"; -import { LOCATION_OPTIONS } from "@/client/features/keywords/locations"; +import { + LOCATION_OPTIONS, + isLabsLocationCode, +} from "@/client/features/keywords/locations"; import type { KeywordResearchControllerState } from "./types"; type Props = { @@ -139,6 +142,49 @@ export function KeywordResearchSearchBar({ controller }: Props) { ) : null; }} + + {(locationField) => + isLabsLocationCode(locationField.state.value) ? ( + + {(field) => ( +
+ +
+ +
+
+ )} + + ) : ( +
+ + + Keyword data for this country comes from Google Ads — search + volume, CPC, and trends are available, but difficulty and + intent are not. + +
+ ) + } + ); diff --git a/src/client/features/keywords/state/keywordControllerActions.ts b/src/client/features/keywords/state/keywordControllerActions.ts index 98c27c2..192a2a5 100644 --- a/src/client/features/keywords/state/keywordControllerActions.ts +++ b/src/client/features/keywords/state/keywordControllerActions.ts @@ -65,12 +65,14 @@ export function buildKeywordSearchKey(params: { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }) { return [ parseKeywordInput(params.keyword).join(""), params.locationCode, params.resultLimit, params.mode, + params.clickstream ? "cs" : "", ].join("|"); } diff --git a/src/client/features/keywords/state/useKeywordResearchController.ts b/src/client/features/keywords/state/useKeywordResearchController.ts index 4718ef1..3cbc1e8 100644 --- a/src/client/features/keywords/state/useKeywordResearchController.ts +++ b/src/client/features/keywords/state/useKeywordResearchController.ts @@ -34,6 +34,7 @@ type OpenKeywordTabInput = { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; export type KeywordResearchControllerInput = { @@ -43,6 +44,7 @@ export type KeywordResearchControllerInput = { hasExplicitLocationCode: boolean; resultLimit: ResultLimit; keywordMode: KeywordMode; + clickstream: boolean; sortField: SortField; sortDir: SortDir; getOpenKeywordTabs?: () => readonly OpenKeywordTabInput[]; @@ -115,6 +117,7 @@ export function useKeywordResearchController( locationCode, resultLimit: input.resultLimit, mode: input.keywordMode, + clickstream: input.clickstream, }, addSearch, ); @@ -127,6 +130,7 @@ export function useKeywordResearchController( locationCode, resultLimit: input.resultLimit, mode: input.keywordMode, + clickstream: input.clickstream, }) : null; diff --git a/src/client/features/rank-tracking/KeywordSuggestionStep.tsx b/src/client/features/rank-tracking/KeywordSuggestionStep.tsx index 0f22d27..8c2ba89 100644 --- a/src/client/features/rank-tracking/KeywordSuggestionStep.tsx +++ b/src/client/features/rank-tracking/KeywordSuggestionStep.tsx @@ -9,6 +9,7 @@ import { Loader2, AlertCircle, X } from "lucide-react"; import { toast } from "sonner"; import { getDomainKeywordSuggestions } from "@/serverFunctions/domain"; import { addTrackingKeywords } from "@/serverFunctions/rank-tracking"; +import { isLabsLocationCode } from "@/client/features/keywords/locations"; import { getStandardErrorMessage } from "@/client/lib/error-messages"; import { AppDataTable, @@ -158,6 +159,9 @@ export function KeywordSuggestionStep({ [], ); + // Ranked-keyword suggestions are Labs-backed; countries served from Google + // Ads keyword data (e.g. Iceland) have no ranking data to suggest from. + const labsSupported = isLabsLocationCode(locationCode); const suggestionsQuery = useQuery({ queryKey: [ "domainKeywordSuggestions", @@ -170,6 +174,7 @@ export function KeywordSuggestionStep({ getDomainKeywordSuggestions({ data: { projectId, domain, locationCode, languageCode }, }), + enabled: labsSupported, }); const data = suggestionsQuery.data ?? []; @@ -238,6 +243,23 @@ export function KeywordSuggestionStep({ ); + if (!labsSupported) { + return ( + <> + {sectionHeader("Add keywords manually")} +
+

+ Ranked-keyword suggestions aren't available for this country. + Continue and add the keywords you want to track manually. +

+ +
+ + ); + } + // Loading state if (suggestionsQuery.isLoading) { return ( diff --git a/src/client/features/search-tabs/SearchTabStrip.tsx b/src/client/features/search-tabs/SearchTabStrip.tsx index 16e1695..e3fa634 100644 --- a/src/client/features/search-tabs/SearchTabStrip.tsx +++ b/src/client/features/search-tabs/SearchTabStrip.tsx @@ -220,6 +220,7 @@ function getSearchTabQueryConfig( locationCode: input.locationCode, resultLimit: input.resultLimit, mode: input.mode, + clickstream: input.clickstream, }); return { diff --git a/src/client/features/search-tabs/types.ts b/src/client/features/search-tabs/types.ts index c85a2c8..ee21d64 100644 --- a/src/client/features/search-tabs/types.ts +++ b/src/client/features/search-tabs/types.ts @@ -23,6 +23,7 @@ export type KeywordSearchTabInput = { locationCode: number; resultLimit: ResultLimit; mode: KeywordMode; + clickstream: boolean; }; export type SearchTabInput = diff --git a/src/client/features/search-tabs/useSearchTabs.ts b/src/client/features/search-tabs/useSearchTabs.ts index 5c7ce9e..45ec299 100644 --- a/src/client/features/search-tabs/useSearchTabs.ts +++ b/src/client/features/search-tabs/useSearchTabs.ts @@ -77,6 +77,8 @@ function parseTabInput(value: unknown): SearchTabInput | null { locationCode: value.locationCode, resultLimit: value.resultLimit, mode: value.mode, + // Tabs persisted before the clickstream toggle existed default to off. + clickstream: value.clickstream === true, }; } diff --git a/src/routes/_project/p/$projectId/keywords.tsx b/src/routes/_project/p/$projectId/keywords.tsx index d51d7de..848a962 100644 --- a/src/routes/_project/p/$projectId/keywords.tsx +++ b/src/routes/_project/p/$projectId/keywords.tsx @@ -47,6 +47,7 @@ function KeywordResearchPageRoute() { hasExplicitLocationCode={search.loc != null} resultLimit={isResultLimit(resultLimit) ? resultLimit : 150} keywordMode={normalizeKeywordMode(keywordMode)} + clickstream={search.cs ?? false} sortField={normalizeSortField(sortField)} sortDir={normalizeSortDir(sortDir)} /> diff --git a/src/server/features/keywords/services/research/research-data.test.ts b/src/server/features/keywords/services/research/research-data.test.ts new file mode 100644 index 0000000..877d174 --- /dev/null +++ b/src/server/features/keywords/services/research/research-data.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it, vi } from "vitest"; + +vi.mock("@/server/lib/dataforseo", () => ({ + createDataforseoClient: vi.fn(), +})); + +import { + KeywordsDataGoogleAdsKeywordsForKeywordsLiveResultInfo, + MonthlySearchesInfo, +} from "dataforseo-client"; +import { mapAdsKeywordItems } from "./research-data"; + +const adsItem = ( + data: ConstructorParameters< + typeof KeywordsDataGoogleAdsKeywordsForKeywordsLiveResultInfo + >[0], +) => new KeywordsDataGoogleAdsKeywordsForKeywordsLiveResultInfo(data); + +describe("mapAdsKeywordItems", () => { + it("maps Google Ads items to research rows without KD/intent", () => { + const rows = mapAdsKeywordItems([ + adsItem({ + keyword: "Hotel Reykjavik", + search_volume: 1300, + cpc: 2.54, + competition: "HIGH", + competition_index: 42, + monthly_searches: [ + new MonthlySearchesInfo({ + year: 2026, + month: 5, + search_volume: 1300, + }), + ], + }), + ]); + + expect(rows).toEqual([ + { + keyword: "hotel reykjavik", + searchVolume: 1300, + trend: [{ year: 2026, month: 5, searchVolume: 1300 }], + cpc: 2.54, + competition: 0.42, + keywordDifficulty: null, + intent: "unknown", + }, + ]); + }); + + it("dedupes case-variant keywords and skips empty ones", () => { + const rows = mapAdsKeywordItems([ + adsItem({ keyword: "northern lights tour", search_volume: 320 }), + adsItem({ keyword: "Northern Lights Tour", search_volume: 320 }), + adsItem({ keyword: undefined }), + ]); + + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + keyword: "northern lights tour", + searchVolume: 320, + competition: null, + cpc: null, + trend: [], + }); + }); +}); diff --git a/src/server/features/keywords/services/research/research-data.ts b/src/server/features/keywords/services/research/research-data.ts index 6af70ea..fe5d3b9 100644 --- a/src/server/features/keywords/services/research/research-data.ts +++ b/src/server/features/keywords/services/research/research-data.ts @@ -1,4 +1,7 @@ -import { type LabsKeywordDataItem } from "@/server/lib/dataforseo"; +import { + type AdsKeywordIdeaItem, + type LabsKeywordDataItem, +} from "@/server/lib/dataforseo"; import type { BillingCustomerContext } from "@/server/billing/subscription"; import { createDataforseoClient } from "@/server/lib/dataforseo"; import { @@ -14,6 +17,7 @@ type FetchResearchRowsParams = { languageCode: string; resultLimit: number; source: KeywordSource; + includeClickstreamData?: boolean; }; function mapKeywordDataItems(items: LabsKeywordDataItem[]): EnrichedKeyword[] { @@ -28,6 +32,8 @@ function mapKeywordDataItems(items: LabsKeywordDataItem[]): EnrichedKeyword[] { if (seen.has(normalized)) continue; seen.add(normalized); + // The clickstream-normalized block only exists when the caller opted into + // clickstream data (it doubles the request cost); prefer it when present. const keywordInfo = item.keyword_info_normalized_with_clickstream ?.search_volume ? item.keyword_info_normalized_with_clickstream @@ -51,6 +57,59 @@ function mapKeywordDataItems(items: LabsKeywordDataItem[]): EnrichedKeyword[] { return rows; } +/** + * Google Ads items carry volume / CPC / paid competition but no keyword + * difficulty or search intent (those are Labs-only). + */ +export function mapAdsKeywordItems( + items: AdsKeywordIdeaItem[], +): EnrichedKeyword[] { + const rows: EnrichedKeyword[] = []; + const seen = new Set(); + + for (const item of items) { + const keyword = item.keyword; + if (!keyword) continue; + + const normalized = normalizeKeyword(keyword); + if (seen.has(normalized)) continue; + seen.add(normalized); + + rows.push({ + keyword: normalized, + searchVolume: item.search_volume ?? null, + trend: (item.monthly_searches ?? []).map((entry) => ({ + year: entry.year ?? 0, + month: entry.month ?? 0, + searchVolume: entry.search_volume ?? 0, + })), + cpc: item.cpc ?? null, + competition: + item.competition_index != null ? item.competition_index / 100 : null, + keywordDifficulty: null, + intent: "unknown", + }); + } + + return rows; +} + +/** Research rows for countries DataForSEO Labs doesn't support. */ +export async function fetchGoogleAdsResearchRows( + params: Omit, + billingCustomer: BillingCustomerContext, +): Promise { + const dataforseo = createDataforseoClient(billingCustomer); + return mapAdsKeywordItems( + await dataforseo.keywords.adsIdeas({ + keyword: params.seedKeyword, + locationCode: params.locationCode, + languageCode: params.languageCode, + limit: params.resultLimit, + }), + ); +} + async function fetchRelatedRows( params: Omit, dataforseo: ReturnType, @@ -61,6 +120,7 @@ async function fetchRelatedRows( languageCode: params.languageCode, limit: params.resultLimit, depth: 3, + includeClickstreamData: params.includeClickstreamData, }); // Related items wrap the keyword payload one level deeper; unwrap and reuse @@ -89,6 +149,7 @@ export async function fetchResearchRowsBySource( locationCode: params.locationCode, languageCode: params.languageCode, limit: params.resultLimit, + includeClickstreamData: params.includeClickstreamData, }), ); } @@ -99,6 +160,7 @@ export async function fetchResearchRowsBySource( locationCode: params.locationCode, languageCode: params.languageCode, limit: params.resultLimit, + includeClickstreamData: params.includeClickstreamData, }), ); } diff --git a/src/server/features/keywords/services/research/research.ts b/src/server/features/keywords/services/research/research.ts index 7308c2d..a8e8a9e 100644 --- a/src/server/features/keywords/services/research/research.ts +++ b/src/server/features/keywords/services/research/research.ts @@ -10,8 +10,12 @@ import { KeywordResearchRepository } from "@/server/features/keywords/repositori import type { KeywordResearchRow } from "@/types/keywords"; import type { ResearchKeywordsInput } from "@/types/schemas/keywords"; import { z } from "zod"; +import { getKeywordDataProvider } from "@/shared/keyword-locations"; import { type EnrichedKeyword, normalizeKeyword } from "./helpers"; -import { fetchResearchRowsBySource } from "./research-data"; +import { + fetchGoogleAdsResearchRows, + fetchResearchRowsBySource, +} from "./research-data"; import { AUTO_KEYWORD_SOURCES, MIN_NON_SEED_FOR_AUTO, @@ -19,10 +23,11 @@ import { hasSufficientCoverage, type KeywordMode, type KeywordSource, + type ResearchSource, } from "./selection"; type SourceAttempt = { - source: KeywordSource; + source: ResearchSource; rowCount: number; nonSeedCount: number; }; @@ -35,7 +40,7 @@ type ResearchDiagnostics = { type ResearchResult = { rows: KeywordResearchRow[]; - source: KeywordSource; + source: ResearchSource; usedFallback: boolean; diagnostics: ResearchDiagnostics; }; @@ -65,14 +70,14 @@ const cachedKeywordRowSchema = z.object({ }); const sourceAttemptSchema = z.object({ - source: z.enum(["related", "suggestions", "ideas"]), + source: z.enum(["related", "suggestions", "ideas", "google_ads"]), rowCount: z.number(), nonSeedCount: z.number(), }); const cachedResultSchema = z.object({ rows: z.array(cachedKeywordRowSchema), - source: z.enum(["related", "suggestions", "ideas"]), + source: z.enum(["related", "suggestions", "ideas", "google_ads"]), usedFallback: z.boolean(), diagnostics: z.object({ requestedMode: z.enum(["auto", "related", "suggestions", "ideas"]), @@ -81,7 +86,9 @@ const cachedResultSchema = z.object({ }), }); -const CACHE_VERSION = 2; +// v3: research volumes are no longer clickstream-refined, and Google-Ads-only +// locations route to keywords_for_keywords. +const CACHE_VERSION = 3; async function fetchRowsFromSource( source: KeywordSource, @@ -96,6 +103,7 @@ async function fetchRowsFromSource( locationCode: input.locationCode, languageCode: input.languageCode, resultLimit: input.resultLimit, + includeClickstreamData: input.clickstream, }, billingCustomer, ); @@ -161,6 +169,39 @@ async function fetchAutoRows( }; } +async function fetchGoogleAdsRows( + input: ResearchKeywordsInput, + seedKeyword: string, + billingCustomer: BillingCustomerContext, +): Promise { + const rows = await fetchGoogleAdsResearchRows( + { + seedKeyword, + locationCode: input.locationCode, + languageCode: input.languageCode, + resultLimit: input.resultLimit, + }, + billingCustomer, + ); + + return { + rows, + source: "google_ads", + usedFallback: false, + diagnostics: { + requestedMode: "auto", + threshold: MIN_NON_SEED_FOR_AUTO, + sourceAttempts: [ + { + source: "google_ads", + rowCount: rows.length, + nonSeedCount: countNonSeedKeywords(rows, seedKeyword), + }, + ], + }, + }; +} + async function fetchManualRows( mode: Exclude, input: ResearchKeywordsInput, @@ -207,6 +248,7 @@ async function buildResearchCacheKey( resultLimit: input.resultLimit, mode, depth: 3, + clickstream: input.clickstream, }); } @@ -244,9 +286,17 @@ export async function research( } const seedKeyword = uniqueKeywords[0]; - const mode = input.mode ?? "auto"; + const provider = getKeywordDataProvider(input.locationCode); + // Labs source modes and clickstream refinement don't exist for + // Google-Ads-served countries; collapse both so equivalent requests share + // one cache entry. + const effectiveInput: ResearchKeywordsInput = + provider === "google_ads" + ? { ...input, mode: "auto", clickstream: false } + : input; + const mode = effectiveInput.mode ?? "auto"; const cacheKey = await buildResearchCacheKey( - input, + effectiveInput, uniqueKeywords, mode, billingCustomer, @@ -263,12 +313,19 @@ export async function research( } const result = - mode === "auto" - ? await fetchAutoRows(input, seedKeyword, billingCustomer) - : await fetchManualRows(mode, input, seedKeyword, billingCustomer); + provider === "google_ads" + ? await fetchGoogleAdsRows(effectiveInput, seedKeyword, billingCustomer) + : mode === "auto" + ? await fetchAutoRows(effectiveInput, seedKeyword, billingCustomer) + : await fetchManualRows( + mode, + effectiveInput, + seedKeyword, + billingCustomer, + ); await setCached(cacheKey, result, CACHE_TTL.researchResult); - persistRows(input, result.rows); + persistRows(effectiveInput, result.rows); return result; } diff --git a/src/server/features/keywords/services/research/selection.ts b/src/server/features/keywords/services/research/selection.ts index 33886ba..e5d00f6 100644 --- a/src/server/features/keywords/services/research/selection.ts +++ b/src/server/features/keywords/services/research/selection.ts @@ -2,6 +2,11 @@ import type { EnrichedKeyword } from "./helpers"; export type KeywordSource = "related" | "suggestions" | "ideas"; export type KeywordMode = "auto" | KeywordSource; +/** + * Where research rows actually came from. "google_ads" is not requestable as + * a mode; it's the automatic source for countries Labs doesn't support. + */ +export type ResearchSource = KeywordSource | "google_ads"; export const AUTO_KEYWORD_SOURCES: KeywordSource[] = [ "related", diff --git a/src/server/features/rank-tracking/services/RankTrackingService.ts b/src/server/features/rank-tracking/services/RankTrackingService.ts index bffb76f..595a925 100644 --- a/src/server/features/rank-tracking/services/RankTrackingService.ts +++ b/src/server/features/rank-tracking/services/RankTrackingService.ts @@ -1,6 +1,7 @@ import { env } from "cloudflare:workers"; import type { BillingCustomerContext } from "@/server/billing/subscription"; import { createDataforseoClient } from "@/server/lib/dataforseo"; +import { getKeywordDataProvider } from "@/shared/keyword-locations"; import { RankTrackingRepository } from "@/server/features/rank-tracking/repositories/RankTrackingRepository"; import { AppError } from "@/server/lib/errors"; import type { @@ -252,13 +253,17 @@ async function refreshKeywordMetrics( const now = new Date().toISOString(); let updated = 0; + // Countries Labs doesn't cover get volume/CPC from Google Ads (no KD). + const useGoogleAds = + getKeywordDataProvider(config.locationCode) === "google_ads"; + for (let i = 0; i < keywords.length; i += KEYWORD_OVERVIEW_BATCH_SIZE) { const batch = keywords.slice(i, i + KEYWORD_OVERVIEW_BATCH_SIZE); - const items = await client.labs.keywordOverview({ + const request = { keywords: batch.map((kw) => kw.keyword), locationCode: config.locationCode, languageCode: config.languageCode, - }); + }; // Build a lookup by lowercase keyword const metricsMap = new Map< @@ -269,13 +274,29 @@ async function refreshKeywordMetrics( cpc: number | null; } >(); - for (const item of items) { - if (!item.keyword) continue; - metricsMap.set(item.keyword.toLowerCase(), { - searchVolume: item.keyword_info?.search_volume ?? null, - keywordDifficulty: item.keyword_properties?.keyword_difficulty ?? null, - cpc: item.keyword_info?.cpc ?? null, + if (useGoogleAds) { + const adsItems = await client.keywords.adsSearchVolume({ + ...request, + creditFeature: "rank_tracking", }); + for (const item of adsItems) { + if (!item.keyword) continue; + metricsMap.set(item.keyword.toLowerCase(), { + searchVolume: item.search_volume ?? null, + keywordDifficulty: null, + cpc: item.cpc ?? null, + }); + } + } else { + for (const item of await client.labs.keywordOverview(request)) { + if (!item.keyword) continue; + metricsMap.set(item.keyword.toLowerCase(), { + searchVolume: item.keyword_info?.search_volume ?? null, + keywordDifficulty: + item.keyword_properties?.keyword_difficulty ?? null, + cpc: item.keyword_info?.cpc ?? null, + }); + } } const updates = batch diff --git a/src/server/lib/dataforseo/client.ts b/src/server/lib/dataforseo/client.ts index 81f48fa..f00967e 100644 --- a/src/server/lib/dataforseo/client.ts +++ b/src/server/lib/dataforseo/client.ts @@ -33,6 +33,10 @@ import { fetchRelevantPages, fetchSerpCompetitors, } from "@/server/lib/dataforseo/labs"; +import { + fetchAdsKeywordIdeas, + fetchAdsSearchVolume, +} from "@/server/lib/dataforseo/google-ads"; import { fetchLiveSerp, fetchLocalSerp, @@ -101,6 +105,9 @@ export function createDataforseoClient(customer: BillingCustomerContext) { related: meter(customer, fetchRelatedKeywords), suggestions: meter(customer, fetchKeywordSuggestions), ideas: meter(customer, fetchKeywordIdeas), + // Google Ads endpoints for countries Labs doesn't support. + adsIdeas: meter(customer, fetchAdsKeywordIdeas), + adsSearchVolume: meter(customer, fetchAdsSearchVolume), }, domain: { rankOverview: meter(customer, fetchDomainRankOverview), diff --git a/src/server/lib/dataforseo/core.ts b/src/server/lib/dataforseo/core.ts index 277d177..493bf9c 100644 --- a/src/server/lib/dataforseo/core.ts +++ b/src/server/lib/dataforseo/core.ts @@ -3,6 +3,7 @@ import { BacklinksApi, BusinessDataApi, DataforseoLabsApi, + KeywordsDataApi, OnPageApi, SerpApi, } from "dataforseo-client"; @@ -119,6 +120,7 @@ function http(classify?: DataforseoErrorClassifier) { // Per-section API factories. Each is created per-request so the auth secret is // read lazily (it lives in the Worker env, not in module scope). export const labsApi = () => new DataforseoLabsApi(API_BASE, http()); +export const keywordsDataApi = () => new KeywordsDataApi(API_BASE, http()); export const serpApi = () => new SerpApi(API_BASE, http()); export const businessDataApi = () => new BusinessDataApi(API_BASE, http()); export const onPageApi = () => new OnPageApi(API_BASE, http()); diff --git a/src/server/lib/dataforseo/google-ads.ts b/src/server/lib/dataforseo/google-ads.ts new file mode 100644 index 0000000..c095b15 --- /dev/null +++ b/src/server/lib/dataforseo/google-ads.ts @@ -0,0 +1,69 @@ +import { + KeywordsDataGoogleAdsKeywordsForKeywordsLiveRequestInfo, + KeywordsDataGoogleAdsSearchVolumeLiveRequestInfo, + type KeywordsDataGoogleAdsKeywordsForKeywordsLiveResultInfo, + type KeywordsDataGoogleAdsSearchVolumeLiveResultInfo, +} from "dataforseo-client"; +import { keywordsDataApi } from "@/server/lib/dataforseo/core"; +import { + assertOk, + buildTaskBilling, + type DataforseoApiResponse, +} from "@/server/lib/dataforseo/envelope"; + +// Google Ads keyword data for countries DataForSEO Labs doesn't cover (see +// specs/0004-keyword-data-source-routing.md). Flat-priced per request; items +// carry volume / CPC / competition but no keyword difficulty or intent. +export type AdsKeywordItem = KeywordsDataGoogleAdsSearchVolumeLiveResultInfo; +export type AdsKeywordIdeaItem = + KeywordsDataGoogleAdsKeywordsForKeywordsLiveResultInfo; + +type KeywordsDataResult = { result?: T[] }; + +function taskItems(task: KeywordsDataResult): T[] { + // keywords_data tasks return keyword items directly in `result` (no nested + // `items` wrapper like Labs). + return task.result ?? []; +} + +export async function fetchAdsSearchVolume(input: { + keywords: string[]; + locationCode: number; + languageCode: string; +}): Promise> { + const response = await keywordsDataApi().googleAdsSearchVolumeLive([ + new KeywordsDataGoogleAdsSearchVolumeLiveRequestInfo({ + keywords: input.keywords, + location_code: input.locationCode, + language_code: input.languageCode, + }), + ]); + const task = assertOk(response); + return { + data: taskItems(task), + billing: buildTaskBilling(task), + }; +} + +export async function fetchAdsKeywordIdeas(input: { + keyword: string; + locationCode: number; + languageCode: string; + limit: number; +}): Promise> { + const response = await keywordsDataApi().googleAdsKeywordsForKeywordsLive([ + new KeywordsDataGoogleAdsKeywordsForKeywordsLiveRequestInfo({ + keywords: [input.keyword], + location_code: input.locationCode, + language_code: input.languageCode, + sort_by: "search_volume", + }), + ]); + const task = assertOk(response); + // The endpoint has no limit parameter (it can return thousands of + // suggestions for one flat fee); truncate to what the caller asked for. + return { + data: taskItems(task).slice(0, input.limit), + billing: buildTaskBilling(task), + }; +} diff --git a/src/server/lib/dataforseo/index.ts b/src/server/lib/dataforseo/index.ts index a960c42..d7a73ff 100644 --- a/src/server/lib/dataforseo/index.ts +++ b/src/server/lib/dataforseo/index.ts @@ -12,6 +12,11 @@ export { type KeywordOverviewItem, } from "@/server/lib/dataforseo/labs"; +export { + type AdsKeywordItem, + type AdsKeywordIdeaItem, +} from "@/server/lib/dataforseo/google-ads"; + export { type SerpLiveItem, type RankCheckResult, diff --git a/src/server/lib/dataforseo/labs.ts b/src/server/lib/dataforseo/labs.ts index 5fb9fc9..2eb97ff 100644 --- a/src/server/lib/dataforseo/labs.ts +++ b/src/server/lib/dataforseo/labs.ts @@ -102,6 +102,7 @@ export async function fetchRelatedKeywords(input: { languageCode: string; limit: number; depth?: number; + includeClickstreamData?: boolean; }): Promise> { const response = await labsApi().googleRelatedKeywordsLive([ new DataforseoLabsGoogleRelatedKeywordsLiveRequestInfo({ @@ -110,7 +111,9 @@ export async function fetchRelatedKeywords(input: { language_code: input.languageCode, limit: input.limit, depth: input.depth ?? 3, - include_clickstream_data: true, + // Clickstream-refined volumes DOUBLE the request cost, so they are + // opt-in — see specs/0004-keyword-data-source-routing.md. + include_clickstream_data: input.includeClickstreamData ?? false, include_serp_info: false, }), ]); @@ -126,6 +129,7 @@ export async function fetchKeywordSuggestions(input: { locationCode: number; languageCode: string; limit: number; + includeClickstreamData?: boolean; }): Promise> { const response = await labsApi().googleKeywordSuggestionsLive([ new DataforseoLabsGoogleKeywordSuggestionsLiveRequestInfo({ @@ -133,7 +137,7 @@ export async function fetchKeywordSuggestions(input: { location_code: input.locationCode, language_code: input.languageCode, limit: input.limit, - include_clickstream_data: true, + include_clickstream_data: input.includeClickstreamData ?? false, include_serp_info: false, include_seed_keyword: true, ignore_synonyms: false, @@ -152,6 +156,7 @@ export async function fetchKeywordIdeas(input: { locationCode: number; languageCode: string; limit: number; + includeClickstreamData?: boolean; }): Promise> { const response = await labsApi().googleKeywordIdeasLive([ new DataforseoLabsGoogleKeywordIdeasLiveRequestInfo({ @@ -159,7 +164,7 @@ export async function fetchKeywordIdeas(input: { location_code: input.locationCode, language_code: input.languageCode, limit: input.limit, - include_clickstream_data: true, + include_clickstream_data: input.includeClickstreamData ?? false, include_serp_info: false, ignore_synonyms: false, closely_variants: false, @@ -274,12 +279,14 @@ export async function fetchKeywordOverview(input: { keywords: string[]; locationCode: number; languageCode: string; + includeClickstreamData?: boolean; }): Promise> { const response = await labsApi().googleKeywordOverviewLive([ new DataforseoLabsGoogleKeywordOverviewLiveRequestInfo({ keywords: input.keywords, location_code: input.locationCode, language_code: input.languageCode, + include_clickstream_data: input.includeClickstreamData ?? false, }), ]); const task = assertOk(response); diff --git a/src/server/mcp/schemas.ts b/src/server/mcp/schemas.ts index db22a6e..dd805d8 100644 --- a/src/server/mcp/schemas.ts +++ b/src/server/mcp/schemas.ts @@ -1,4 +1,6 @@ import { z } from "zod"; +import { AppError } from "@/server/lib/errors"; +import { getKeywordDataProvider } from "@/shared/keyword-locations"; export const DEFAULT_LOCATION_CODE = 2840; export const DEFAULT_LANGUAGE_CODE = "en"; @@ -15,9 +17,22 @@ export const locationCodeSchema = z .int() .positive() .describe( - "DataForSEO location code. Defaults to 2840 (United States). See dataforseo.com/help-center/locations.", + "DataForSEO location code. Defaults to 2840 (United States). See dataforseo.com/help-center/locations. Some countries (e.g. Iceland, 2352) are served from Google Ads data: keyword volume/CPC/trends work, but keyword difficulty, search intent, and domain analytics are unavailable.", ); +/** + * Guards Labs-backed tools (domain analytics) against locations we serve + * from Google Ads keyword data only. + */ +export function assertLabsLocationCode(locationCode: number | undefined) { + if (locationCode != null && getKeywordDataProvider(locationCode) !== "labs") { + throw new AppError( + "VALIDATION_ERROR", + "Domain analytics is not available for this country. Keyword research and rank tracking work; domain-level data is limited to DataForSEO Labs locations.", + ); + } +} + export const languageCodeSchema = z .string() .min(2) diff --git a/src/server/mcp/tools/dataforseo-research-tools.google-ads.test.ts b/src/server/mcp/tools/dataforseo-research-tools.google-ads.test.ts new file mode 100644 index 0000000..f12a8d1 --- /dev/null +++ b/src/server/mcp/tools/dataforseo-research-tools.google-ads.test.ts @@ -0,0 +1,157 @@ +import type { AuthInfo } from "@modelcontextprotocol/sdk/server/auth/types.js"; +import type { ToolExtra } from "@/server/mcp/context"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { z } from "zod"; +import { MCP_AUTH_CONTEXT_PROP } from "@/server/mcp/context"; + +const mocks = vi.hoisted(() => ({ + createDataforseoClient: vi.fn(), + getProjectForOrganization: vi.fn(), +})); + +vi.mock("cloudflare:workers", () => ({ + env: {}, +})); + +vi.mock("@/server/lib/dataforseo", () => ({ + createDataforseoClient: mocks.createDataforseoClient, +})); + +vi.mock("@/server/features/projects/services/ProjectService", () => ({ + ProjectService: { + getProjectForOrganization: mocks.getProjectForOrganization, + }, +})); + +const authContext = { + userId: "user_123", + userEmail: "alice@example.com", + organizationId: "org_123", + clientId: "client_123", + scopes: ["mcp"], + audience: "https://open-seo.test/mcp", + subject: "user_123", + baseUrl: "https://open-seo.test", +}; + +const toolExtra: ToolExtra = { + signal: new AbortController().signal, + requestId: 1, + sendNotification: vi.fn(), + sendRequest: vi.fn(), + authInfo: { + token: "token", + clientId: "client_123", + scopes: ["mcp"], + resource: new URL("https://open-seo.test/mcp"), + extra: { [MCP_AUTH_CONTEXT_PROP]: authContext }, + } satisfies AuthInfo, +}; + +describe("get_keyword_metrics for Google-Ads-only locations", () => { + beforeEach(() => { + vi.resetModules(); + mocks.createDataforseoClient.mockReset(); + mocks.getProjectForOrganization.mockReset(); + mocks.getProjectForOrganization.mockResolvedValue({ id: "project_1" }); + }); + + it("serves Iceland from adsSearchVolume without KD/intent", async () => { + const keywordOverview = vi.fn(); + const adsSearchVolume = vi.fn().mockResolvedValue([ + { + keyword: "hotel reykjavik", + search_volume: 1300, + cpc: 2.54, + competition: "HIGH", + competition_index: 42, + monthly_searches: [{ year: 2026, month: 5, search_volume: 1300 }], + }, + ]); + + mocks.createDataforseoClient.mockReturnValue({ + labs: { keywordOverview }, + keywords: { adsSearchVolume }, + }); + const { getKeywordMetricsTool } = + await import("./dataforseo-research-tools"); + + const result = await getKeywordMetricsTool.handler( + { + projectId: "project_1", + keywords: ["hotel reykjavik"], + // Iceland is not supported by DataForSEO Labs. + locationCode: 2352, + languageCode: "is", + }, + toolExtra, + ); + + expect(keywordOverview).not.toHaveBeenCalled(); + expect(adsSearchVolume).toHaveBeenCalledWith( + expect.objectContaining({ + keywords: ["hotel reykjavik"], + locationCode: 2352, + languageCode: "is", + creditFeature: "keyword_research", + }), + ); + const rows = z + .object({ keywords: z.array(z.record(z.string(), z.unknown())) }) + .passthrough() + .parse(result.structuredContent).keywords; + expect(rows[0]).toMatchObject({ + keyword: "hotel reykjavik", + search_volume: 1300, + keyword_difficulty: null, + main_intent: null, + cpc: 2.54, + competition: 0.42, + competition_level: "HIGH", + }); + }); + + it("passes the clickstream opt-in to Labs and prefers refined volumes", async () => { + const keywordOverview = vi.fn().mockResolvedValue([ + { + keyword: "seo tools", + keyword_info: { + search_volume: 10000, + monthly_searches: [{ year: 2026, month: 5, search_volume: 10000 }], + }, + keyword_info_normalized_with_clickstream: { + search_volume: 6400, + monthly_searches: [{ year: 2026, month: 5, search_volume: 6400 }], + }, + }, + ]); + + mocks.createDataforseoClient.mockReturnValue({ + labs: { keywordOverview }, + }); + const { getKeywordMetricsTool } = + await import("./dataforseo-research-tools"); + + const result = await getKeywordMetricsTool.handler( + { + projectId: "project_1", + keywords: ["seo tools"], + includeClickstreamData: true, + }, + toolExtra, + ); + + expect(keywordOverview).toHaveBeenCalledWith( + expect.objectContaining({ includeClickstreamData: true }), + ); + const rows = z + .object({ keywords: z.array(z.record(z.string(), z.unknown())) }) + .passthrough() + .parse(result.structuredContent).keywords; + expect(rows[0]).toMatchObject({ + keyword: "seo tools", + search_volume: 6400, + monthly_searches: [{ year: 2026, month: 5, search_volume: 6400 }], + }); + }); +}); diff --git a/src/server/mcp/tools/dataforseo-research-tools.ts b/src/server/mcp/tools/dataforseo-research-tools.ts index dba9a57..3616649 100644 --- a/src/server/mcp/tools/dataforseo-research-tools.ts +++ b/src/server/mcp/tools/dataforseo-research-tools.ts @@ -2,8 +2,10 @@ import { z } from "zod"; import { createDataforseoClient, + type AdsKeywordItem, type KeywordOverviewItem, } from "@/server/lib/dataforseo"; +import { getKeywordDataProvider } from "@/shared/keyword-locations"; import { buildProjectMeta } from "@/server/mcp/context"; import { mcpResponse } from "@/server/mcp/formatters"; import { @@ -303,6 +305,12 @@ const getKeywordMetricsInputSchema = { .boolean() .optional() .describe("Include monthly search-volume trend rows. Defaults to true."), + includeClickstreamData: z + .boolean() + .optional() + .describe( + "Refine search volumes with clickstream data, which disaggregates Google Ads' grouped close-variant volumes (plurals/misspellings). DOUBLES the credit cost of the call. Default false. No effect for countries served from Google Ads data.", + ), sortBy: keywordMetricsSortSchema .optional() .describe("Sort order for returned rows. Defaults to search_volume."), @@ -457,20 +465,41 @@ function sortCompetitors( function normalizeKeywordOverview(item: KeywordOverviewItem) { const info = item.keyword_info; + // Only present when the caller opted into clickstream-refined volumes. + const clickstreamInfo = item.keyword_info_normalized_with_clickstream; return { keyword: item.keyword, - search_volume: info?.search_volume ?? null, + search_volume: + clickstreamInfo?.search_volume ?? info?.search_volume ?? null, keyword_difficulty: item.keyword_properties?.keyword_difficulty ?? null, main_intent: item.search_intent_info?.main_intent ?? null, cpc: info?.cpc ?? null, competition: info?.competition ?? null, competition_level: info?.competition_level ?? null, - monthly_searches: info?.monthly_searches ?? null, + monthly_searches: + (clickstreamInfo?.search_volume + ? clickstreamInfo.monthly_searches + : info?.monthly_searches) ?? null, }; } type KeywordMetricRow = ReturnType; +// Google Ads items (countries Labs doesn't cover) have no difficulty/intent. +function normalizeAdsKeyword(item: AdsKeywordItem): KeywordMetricRow { + return { + keyword: item.keyword, + search_volume: item.search_volume ?? null, + keyword_difficulty: null, + main_intent: null, + cpc: item.cpc ?? null, + competition: + item.competition_index != null ? item.competition_index / 100 : null, + competition_level: item.competition ?? null, + monthly_searches: item.monthly_searches ?? null, + }; +} + function sortKeywordMetricRows( rows: KeywordMetricRow[], sortBy: NonNullable = "search_volume", @@ -722,7 +751,7 @@ export const getKeywordMetricsTool = { config: { title: "Get keyword metrics", description: - "Hydrate up to 700 known keywords with search volume, keyword difficulty (KD), search intent, CPC, competition, and monthly trends in a single call. Use it to score candidate or known keywords — including Search Console striking-distance queries — by real demand and ranking difficulty. Charges credits.", + "Hydrate up to 700 known keywords with search volume, keyword difficulty (KD), search intent, CPC, competition, and monthly trends in a single call. Use it to score candidate or known keywords — including Search Console striking-distance queries — by real demand and ranking difficulty. For countries served from Google Ads data (e.g. Iceland), KD and intent are null. Charges credits.", inputSchema: getKeywordMetricsInputSchema, outputSchema: { keywords: z.array(looseObjectOutputSchema), @@ -736,14 +765,29 @@ export const getKeywordMetricsTool = { }, handler: withMcpProjectAuth(async (args: GetKeywordMetricsArgs, context) => { const client = createDataforseoClient(context.billing); - const items = await client.labs.keywordOverview({ - keywords: args.keywords, - locationCode: args.locationCode ?? DEFAULT_LOCATION_CODE, - languageCode: args.languageCode ?? DEFAULT_LANGUAGE_CODE, - creditFeature: "keyword_research", - }); + const locationCode = args.locationCode ?? DEFAULT_LOCATION_CODE; + const languageCode = args.languageCode ?? DEFAULT_LANGUAGE_CODE; + const normalized = + getKeywordDataProvider(locationCode) === "google_ads" + ? ( + await client.keywords.adsSearchVolume({ + keywords: args.keywords, + locationCode, + languageCode, + creditFeature: "keyword_research", + }) + ).map(normalizeAdsKeyword) + : ( + await client.labs.keywordOverview({ + keywords: args.keywords, + locationCode, + languageCode, + includeClickstreamData: args.includeClickstreamData ?? false, + creditFeature: "keyword_research", + }) + ).map(normalizeKeywordOverview); const rows = sortKeywordMetricRows( - items.map(normalizeKeywordOverview), + normalized, args.sortBy ?? "search_volume", ).map((row) => args.includeMonthlyTrends === false diff --git a/src/server/mcp/tools/get-domain-keyword-suggestions.ts b/src/server/mcp/tools/get-domain-keyword-suggestions.ts index b451425..cc3b722 100644 --- a/src/server/mcp/tools/get-domain-keyword-suggestions.ts +++ b/src/server/mcp/tools/get-domain-keyword-suggestions.ts @@ -10,6 +10,7 @@ import { withMcpProjectAuth } from "@/server/mcp/project-auth"; import { DEFAULT_LANGUAGE_CODE, DEFAULT_LOCATION_CODE, + assertLabsLocationCode, languageCodeSchema, locationCodeSchema, projectIdSchema, @@ -45,6 +46,7 @@ export const getDomainKeywordSuggestionsTool = { }, }, handler: withMcpProjectAuth(async (args: Args, context) => { + assertLabsLocationCode(args.locationCode); const keywords = await DomainService.getSuggestedKeywords( { domain: args.domain, diff --git a/src/server/mcp/tools/get-domain-overview.ts b/src/server/mcp/tools/get-domain-overview.ts index f2be86f..c0b2263 100644 --- a/src/server/mcp/tools/get-domain-overview.ts +++ b/src/server/mcp/tools/get-domain-overview.ts @@ -7,6 +7,7 @@ import { withMcpProjectAuth } from "@/server/mcp/project-auth"; import { DEFAULT_LANGUAGE_CODE, DEFAULT_LOCATION_CODE, + assertLabsLocationCode, languageCodeSchema, locationCodeSchema, projectIdSchema, @@ -50,6 +51,7 @@ export const getDomainOverviewTool = { }, }, handler: withMcpProjectAuth(async (args: Args, context) => { + assertLabsLocationCode(args.locationCode); const result = await DomainService.getOverview( { projectId: args.projectId, diff --git a/src/server/mcp/tools/research-keywords.ts b/src/server/mcp/tools/research-keywords.ts index f04e66d..268e0a1 100644 --- a/src/server/mcp/tools/research-keywords.ts +++ b/src/server/mcp/tools/research-keywords.ts @@ -34,6 +34,12 @@ const inputSchema = { .union([z.literal(150), z.literal(300), z.literal(500)]) .optional() .describe("Max keywords returned per seed. Defaults to 150."), + includeClickstreamData: z + .boolean() + .optional() + .describe( + "Refine search volumes with clickstream data, which disaggregates Google Ads' grouped close-variant volumes (plurals/misspellings). DOUBLES the credit cost of each seed. Default false (standard Google-Ads-derived volumes). No effect for countries served from Google Ads data.", + ), } as const; type Args = z.infer>; @@ -43,7 +49,7 @@ export const researchKeywordsTool = { config: { title: "Research keywords (bulk)", description: - "Research keyword data (search volume, difficulty, CPC, related ideas) for 1-5 seed keywords in one call. Charges credits per seed (~50-200 credits each, varies by source). Returns per-seed results — a single bad seed won't fail the batch.", + "Research keyword data (search volume, difficulty, CPC, related ideas) for 1-5 seed keywords in one call. Charges credits per seed (~30-100 credits each, varies by source; flat ~96 for countries served from Google Ads data, where difficulty/intent are unavailable). Returns per-seed results — a single bad seed won't fail the batch.", inputSchema, outputSchema: { results: z.array( @@ -87,6 +93,7 @@ export const researchKeywordsTool = { languageCode: item.languageCode ?? DEFAULT_LANGUAGE_CODE, resultLimit: args.resultLimit ?? 150, mode: "auto", + clickstream: args.includeClickstreamData ?? false, }, context.billing, ); diff --git a/src/shared/keyword-locations.test.ts b/src/shared/keyword-locations.test.ts new file mode 100644 index 0000000..1a30dac --- /dev/null +++ b/src/shared/keyword-locations.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from "vitest"; +import { + LABS_LOCATION_OPTIONS, + LOCATION_OPTIONS, + getKeywordDataProvider, + getLanguageCode, + isLabsLocationCode, + isSupportedLocationCode, +} from "./keyword-locations"; + +describe("keyword locations", () => { + it("routes Labs-supported countries to labs", () => { + expect(getKeywordDataProvider(2840)).toBe("labs"); // US + expect(getKeywordDataProvider(2826)).toBe("labs"); // UK + }); + + it("routes Google-Ads-only countries to google_ads", () => { + expect(getKeywordDataProvider(2352)).toBe("google_ads"); // Iceland + expect(isSupportedLocationCode(2352)).toBe(true); + expect(isLabsLocationCode(2352)).toBe(false); + expect(getLanguageCode(2352)).toBe("is"); + }); + + it("falls back to labs for unknown codes (Labs rejects them upstream)", () => { + expect(getKeywordDataProvider(999999)).toBe("labs"); + expect(isSupportedLocationCode(999999)).toBe(false); + }); + + it("excludes every Google-Ads-only country from the Labs picker", () => { + const adsOnly = LOCATION_OPTIONS.filter((option) => option.googleAdsOnly); + expect(adsOnly.length).toBeGreaterThan(0); + const labsCodes = new Set( + LABS_LOCATION_OPTIONS.map((option) => option.code), + ); + for (const option of adsOnly) { + expect(labsCodes.has(option.code)).toBe(false); + } + expect(LABS_LOCATION_OPTIONS.length + adsOnly.length).toBe( + LOCATION_OPTIONS.length, + ); + }); + + it("keeps the picker sorted alphabetically with unique codes", () => { + const labels = LOCATION_OPTIONS.map((option) => option.label); + expect(labels).toEqual(labels.toSorted((a, b) => a.localeCompare(b))); + const codes = LOCATION_OPTIONS.map((option) => option.code); + expect(new Set(codes).size).toBe(codes.length); + }); +}); diff --git a/src/shared/keyword-locations.ts b/src/shared/keyword-locations.ts new file mode 100644 index 0000000..01e8749 --- /dev/null +++ b/src/shared/keyword-locations.ts @@ -0,0 +1,538 @@ +/* eslint-disable max-lines -- country data table */ +/** + * Supported keyword-data countries and their data provider. + * + * Default provider is DataForSEO Labs (94 countries; source: + * https://api.dataforseo.com/v3/dataforseo_labs/locations_and_languages). + * Countries Labs does not cover are marked `googleAdsOnly` and are served by + * the DataForSEO Keywords Data API (Google Ads endpoints), which covers the + * full Google geotarget list — see specs/0004-keyword-data-source-routing.md. + * Google-Ads-only rows have no keyword difficulty or search intent. + * + * For countries with multiple Google-supported languages, we pick the + * language with the largest keyword corpus (the primary search market) + * as the default. The APIs accept a single location_code + language_code + * pair per request, so we expose one entry per country. Language codes for + * googleAdsOnly entries must exist in BOTH the Google Ads and SERP language + * lists (rank tracking shares this picker and uses the SERP API). + * + * Entries are sorted alphabetically by country name; pick US as the + * product-wide default via DEFAULT_LOCATION_CODE below. + */ +export const DEFAULT_LOCATION_CODE = 2840; + +type KeywordDataProvider = "labs" | "google_ads"; + +type LocationOption = { + code: number; + label: string; + shortLabel: string; + languageCode: string; + /** Set when DataForSEO Labs does not support this country. */ + googleAdsOnly?: true; +}; + +export const LOCATION_OPTIONS: readonly LocationOption[] = [ + { code: 2008, label: "Albania", shortLabel: "AL", languageCode: "sq" }, + { code: 2012, label: "Algeria", shortLabel: "DZ", languageCode: "fr" }, + { + code: 2020, + label: "Andorra", + shortLabel: "AD", + languageCode: "ca", + googleAdsOnly: true, + }, + { code: 2024, label: "Angola", shortLabel: "AO", languageCode: "pt" }, + { code: 2032, label: "Argentina", shortLabel: "AR", languageCode: "es" }, + { code: 2051, label: "Armenia", shortLabel: "AM", languageCode: "hy" }, + { code: 2036, label: "Australia", shortLabel: "AU", languageCode: "en" }, + { code: 2040, label: "Austria", shortLabel: "AT", languageCode: "de" }, + { code: 2031, label: "Azerbaijan", shortLabel: "AZ", languageCode: "az" }, + { + code: 2044, + label: "Bahamas", + shortLabel: "BS", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2048, label: "Bahrain", shortLabel: "BH", languageCode: "ar" }, + { code: 2050, label: "Bangladesh", shortLabel: "BD", languageCode: "bn" }, + { + code: 2052, + label: "Barbados", + shortLabel: "BB", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2056, label: "Belgium", shortLabel: "BE", languageCode: "nl" }, + { + code: 2084, + label: "Belize", + shortLabel: "BZ", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2068, label: "Bolivia", shortLabel: "BO", languageCode: "es" }, + { + code: 2070, + label: "Bosnia and Herzegovina", + shortLabel: "BA", + languageCode: "bs", + }, + { + code: 2072, + label: "Botswana", + shortLabel: "BW", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2076, label: "Brazil", shortLabel: "BR", languageCode: "pt" }, + { + code: 2096, + label: "Brunei", + shortLabel: "BN", + languageCode: "ms", + googleAdsOnly: true, + }, + { code: 2100, label: "Bulgaria", shortLabel: "BG", languageCode: "bg" }, + { code: 2854, label: "Burkina Faso", shortLabel: "BF", languageCode: "fr" }, + { code: 2116, label: "Cambodia", shortLabel: "KH", languageCode: "en" }, + { code: 2120, label: "Cameroon", shortLabel: "CM", languageCode: "fr" }, + { code: 2124, label: "Canada", shortLabel: "CA", languageCode: "en" }, + { code: 2152, label: "Chile", shortLabel: "CL", languageCode: "es" }, + { code: 2170, label: "Colombia", shortLabel: "CO", languageCode: "es" }, + { code: 2188, label: "Costa Rica", shortLabel: "CR", languageCode: "es" }, + { code: 2384, label: "Cote d'Ivoire", shortLabel: "CI", languageCode: "fr" }, + { code: 2191, label: "Croatia", shortLabel: "HR", languageCode: "hr" }, + { code: 2196, label: "Cyprus", shortLabel: "CY", languageCode: "el" }, + { code: 2203, label: "Czechia", shortLabel: "CZ", languageCode: "cs" }, + { code: 2208, label: "Denmark", shortLabel: "DK", languageCode: "da" }, + { + code: 2214, + label: "Dominican Republic", + shortLabel: "DO", + languageCode: "es", + googleAdsOnly: true, + }, + { code: 2218, label: "Ecuador", shortLabel: "EC", languageCode: "es" }, + { code: 2818, label: "Egypt", shortLabel: "EG", languageCode: "ar" }, + { code: 2222, label: "El Salvador", shortLabel: "SV", languageCode: "es" }, + { code: 2233, label: "Estonia", shortLabel: "EE", languageCode: "et" }, + { + code: 2231, + label: "Ethiopia", + shortLabel: "ET", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2242, + label: "Fiji", + shortLabel: "FJ", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2246, label: "Finland", shortLabel: "FI", languageCode: "fi" }, + { code: 2250, label: "France", shortLabel: "FR", languageCode: "fr" }, + { + code: 2268, + label: "Georgia", + shortLabel: "GE", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2276, label: "Germany", shortLabel: "DE", languageCode: "de" }, + { code: 2288, label: "Ghana", shortLabel: "GH", languageCode: "en" }, + { code: 2300, label: "Greece", shortLabel: "GR", languageCode: "el" }, + { code: 2320, label: "Guatemala", shortLabel: "GT", languageCode: "es" }, + { + code: 2831, + label: "Guernsey", + shortLabel: "GG", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2328, + label: "Guyana", + shortLabel: "GY", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2332, + label: "Haiti", + shortLabel: "HT", + languageCode: "fr", + googleAdsOnly: true, + }, + { + code: 2340, + label: "Honduras", + shortLabel: "HN", + languageCode: "es", + googleAdsOnly: true, + }, + { code: 2344, label: "Hong Kong", shortLabel: "HK", languageCode: "zh-TW" }, + { code: 2348, label: "Hungary", shortLabel: "HU", languageCode: "hu" }, + { + code: 2352, + label: "Iceland", + shortLabel: "IS", + languageCode: "is", + googleAdsOnly: true, + }, + { code: 2356, label: "India", shortLabel: "IN", languageCode: "en" }, + { code: 2360, label: "Indonesia", shortLabel: "ID", languageCode: "id" }, + { + code: 2368, + label: "Iraq", + shortLabel: "IQ", + languageCode: "ar", + googleAdsOnly: true, + }, + { code: 2372, label: "Ireland", shortLabel: "IE", languageCode: "en" }, + { + code: 2833, + label: "Isle of Man", + shortLabel: "IM", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2376, label: "Israel", shortLabel: "IL", languageCode: "he" }, + { code: 2380, label: "Italy", shortLabel: "IT", languageCode: "it" }, + { + code: 2388, + label: "Jamaica", + shortLabel: "JM", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2392, label: "Japan", shortLabel: "JP", languageCode: "ja" }, + { + code: 2832, + label: "Jersey", + shortLabel: "JE", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2400, label: "Jordan", shortLabel: "JO", languageCode: "ar" }, + { code: 2398, label: "Kazakhstan", shortLabel: "KZ", languageCode: "ru" }, + { code: 2404, label: "Kenya", shortLabel: "KE", languageCode: "en" }, + { + code: 2414, + label: "Kuwait", + shortLabel: "KW", + languageCode: "ar", + googleAdsOnly: true, + }, + { + code: 2417, + label: "Kyrgyzstan", + shortLabel: "KG", + languageCode: "ru", + googleAdsOnly: true, + }, + { + code: 2418, + label: "Laos", + shortLabel: "LA", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2428, label: "Latvia", shortLabel: "LV", languageCode: "lv" }, + { + code: 2422, + label: "Lebanon", + shortLabel: "LB", + languageCode: "ar", + googleAdsOnly: true, + }, + { + code: 2438, + label: "Liechtenstein", + shortLabel: "LI", + languageCode: "de", + googleAdsOnly: true, + }, + { code: 2440, label: "Lithuania", shortLabel: "LT", languageCode: "lt" }, + { + code: 2442, + label: "Luxembourg", + shortLabel: "LU", + languageCode: "fr", + googleAdsOnly: true, + }, + { + code: 2450, + label: "Madagascar", + shortLabel: "MG", + languageCode: "fr", + googleAdsOnly: true, + }, + { + code: 2454, + label: "Malawi", + shortLabel: "MW", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2458, label: "Malaysia", shortLabel: "MY", languageCode: "en" }, + { + code: 2462, + label: "Maldives", + shortLabel: "MV", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2470, label: "Malta", shortLabel: "MT", languageCode: "en" }, + { + code: 2480, + label: "Mauritius", + shortLabel: "MU", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2484, label: "Mexico", shortLabel: "MX", languageCode: "es" }, + { code: 2498, label: "Moldova", shortLabel: "MD", languageCode: "ro" }, + { code: 2492, label: "Monaco", shortLabel: "MC", languageCode: "fr" }, + { + code: 2496, + label: "Mongolia", + shortLabel: "MN", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2499, + label: "Montenegro", + shortLabel: "ME", + languageCode: "sr", + googleAdsOnly: true, + }, + { code: 2504, label: "Morocco", shortLabel: "MA", languageCode: "ar" }, + { + code: 2508, + label: "Mozambique", + shortLabel: "MZ", + languageCode: "pt", + googleAdsOnly: true, + }, + { + code: 2104, + label: "Myanmar (Burma)", + shortLabel: "MM", + languageCode: "en", + }, + { + code: 2516, + label: "Namibia", + shortLabel: "NA", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2524, + label: "Nepal", + shortLabel: "NP", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2528, label: "Netherlands", shortLabel: "NL", languageCode: "nl" }, + { code: 2554, label: "New Zealand", shortLabel: "NZ", languageCode: "en" }, + { code: 2558, label: "Nicaragua", shortLabel: "NI", languageCode: "es" }, + { code: 2566, label: "Nigeria", shortLabel: "NG", languageCode: "en" }, + { + code: 2807, + label: "North Macedonia", + shortLabel: "MK", + languageCode: "mk", + }, + { code: 2578, label: "Norway", shortLabel: "NO", languageCode: "nb" }, + { + code: 2512, + label: "Oman", + shortLabel: "OM", + languageCode: "ar", + googleAdsOnly: true, + }, + { code: 2586, label: "Pakistan", shortLabel: "PK", languageCode: "en" }, + { code: 2591, label: "Panama", shortLabel: "PA", languageCode: "es" }, + { + code: 2598, + label: "Papua New Guinea", + shortLabel: "PG", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2600, label: "Paraguay", shortLabel: "PY", languageCode: "es" }, + { code: 2604, label: "Peru", shortLabel: "PE", languageCode: "es" }, + { code: 2608, label: "Philippines", shortLabel: "PH", languageCode: "en" }, + { code: 2616, label: "Poland", shortLabel: "PL", languageCode: "pl" }, + { code: 2620, label: "Portugal", shortLabel: "PT", languageCode: "pt" }, + { + code: 2634, + label: "Qatar", + shortLabel: "QA", + languageCode: "ar", + googleAdsOnly: true, + }, + { code: 2642, label: "Romania", shortLabel: "RO", languageCode: "ro" }, + { + code: 2646, + label: "Rwanda", + shortLabel: "RW", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2674, + label: "San Marino", + shortLabel: "SM", + languageCode: "it", + googleAdsOnly: true, + }, + { code: 2682, label: "Saudi Arabia", shortLabel: "SA", languageCode: "ar" }, + { code: 2686, label: "Senegal", shortLabel: "SN", languageCode: "fr" }, + { code: 2688, label: "Serbia", shortLabel: "RS", languageCode: "sr" }, + { code: 2702, label: "Singapore", shortLabel: "SG", languageCode: "en" }, + { code: 2703, label: "Slovakia", shortLabel: "SK", languageCode: "sk" }, + { code: 2705, label: "Slovenia", shortLabel: "SI", languageCode: "sl" }, + { code: 2710, label: "South Africa", shortLabel: "ZA", languageCode: "en" }, + { code: 2410, label: "South Korea", shortLabel: "KR", languageCode: "ko" }, + { code: 2724, label: "Spain", shortLabel: "ES", languageCode: "es" }, + { code: 2144, label: "Sri Lanka", shortLabel: "LK", languageCode: "en" }, + { + code: 2740, + label: "Suriname", + shortLabel: "SR", + languageCode: "nl", + googleAdsOnly: true, + }, + { code: 2752, label: "Sweden", shortLabel: "SE", languageCode: "sv" }, + { code: 2756, label: "Switzerland", shortLabel: "CH", languageCode: "de" }, + { code: 2158, label: "Taiwan", shortLabel: "TW", languageCode: "zh-TW" }, + { + code: 2762, + label: "Tajikistan", + shortLabel: "TJ", + languageCode: "ru", + googleAdsOnly: true, + }, + { + code: 2834, + label: "Tanzania", + shortLabel: "TZ", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2764, label: "Thailand", shortLabel: "TH", languageCode: "th" }, + { + code: 2780, + label: "Trinidad and Tobago", + shortLabel: "TT", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2788, label: "Tunisia", shortLabel: "TN", languageCode: "ar" }, + { code: 2792, label: "Turkiye", shortLabel: "TR", languageCode: "tr" }, + { + code: 2795, + label: "Turkmenistan", + shortLabel: "TM", + languageCode: "ru", + googleAdsOnly: true, + }, + { + code: 2800, + label: "Uganda", + shortLabel: "UG", + languageCode: "en", + googleAdsOnly: true, + }, + { code: 2804, label: "Ukraine", shortLabel: "UA", languageCode: "uk" }, + { + code: 2784, + label: "United Arab Emirates", + shortLabel: "AE", + languageCode: "en", + }, + { + code: 2826, + label: "United Kingdom", + shortLabel: "UK", + languageCode: "en", + }, + { code: 2840, label: "United States", shortLabel: "US", languageCode: "en" }, + { code: 2858, label: "Uruguay", shortLabel: "UY", languageCode: "es" }, + { + code: 2860, + label: "Uzbekistan", + shortLabel: "UZ", + languageCode: "ru", + googleAdsOnly: true, + }, + { code: 2862, label: "Venezuela", shortLabel: "VE", languageCode: "es" }, + { code: 2704, label: "Vietnam", shortLabel: "VN", languageCode: "vi" }, + { + code: 2894, + label: "Zambia", + shortLabel: "ZM", + languageCode: "en", + googleAdsOnly: true, + }, + { + code: 2716, + label: "Zimbabwe", + shortLabel: "ZW", + languageCode: "en", + googleAdsOnly: true, + }, +] as const; + +/** Countries usable by DataForSEO Labs features (domain overview etc.). */ +export const LABS_LOCATION_OPTIONS = LOCATION_OPTIONS.filter( + (option) => !option.googleAdsOnly, +); + +const LOCATION_CODES = new Set( + LOCATION_OPTIONS.map((option) => option.code), +); + +const LABS_LOCATION_CODES = new Set( + LABS_LOCATION_OPTIONS.map((option) => option.code), +); + +export const LOCATIONS: Record = Object.fromEntries( + LOCATION_OPTIONS.map((option) => [option.code, option.shortLabel]), +); + +const LOCATION_LANGUAGE: Record = Object.fromEntries( + LOCATION_OPTIONS.map((option) => [option.code, option.languageCode]), +); + +export function getLanguageCode(locationCode: number): string { + return LOCATION_LANGUAGE[locationCode] ?? "en"; +} + +export function isSupportedLocationCode(locationCode: number): boolean { + return LOCATION_CODES.has(locationCode); +} + +export function isLabsLocationCode(locationCode: number): boolean { + return LABS_LOCATION_CODES.has(locationCode); +} + +/** + * Which DataForSEO API serves keyword data for this location. Unknown codes + * fall back to Labs so behavior for arbitrary codes is unchanged (Labs + * rejects unsupported locations with its own error). + */ +export function getKeywordDataProvider( + locationCode: number, +): KeywordDataProvider { + return LOCATION_CODES.has(locationCode) && + !LABS_LOCATION_CODES.has(locationCode) + ? "google_ads" + : "labs"; +} diff --git a/src/types/schemas/domain.ts b/src/types/schemas/domain.ts index ac3e30f..d967b50 100644 --- a/src/types/schemas/domain.ts +++ b/src/types/schemas/domain.ts @@ -51,7 +51,7 @@ export const domainField = z } }); -const booleanSearchParamSchema = z +export const booleanSearchParamSchema = z .union([z.boolean(), z.enum(["true", "false"])]) .transform((value) => value === true || value === "true"); diff --git a/src/types/schemas/keywords.ts b/src/types/schemas/keywords.ts index 1bc30ed..639b532 100644 --- a/src/types/schemas/keywords.ts +++ b/src/types/schemas/keywords.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import { TAG_COLOR_KEYS } from "@/shared/tag-colors"; +import { booleanSearchParamSchema } from "@/types/schemas/domain"; const savedKeywordTagSchema = z.string().trim().min(1).max(64); const tagColorSchema = z.enum(TAG_COLOR_KEYS); @@ -26,6 +27,8 @@ export const researchKeywordsSchema = z.object({ .enum(["auto", "related", "suggestions", "ideas"]) .optional() .default("auto"), + // Clickstream-refined volumes double the DataForSEO request cost; opt-in. + clickstream: z.boolean().optional().default(false), }); export const saveKeywordsSchema = z @@ -184,6 +187,7 @@ export const keywordsSearchSchema = z.object({ loc: z.coerce.number().int().positive().optional(), kLimit: z.union([z.literal(150), z.literal(300), z.literal(500)]).optional(), mode: z.enum(keywordModes).optional(), + cs: booleanSearchParamSchema.optional(), sort: z.enum(keywordSortFields).optional(), order: z.enum(sortDirs).optional(), minVol: z.string().optional(),