Skip to content

Extend resolution search labels and GeoJSON annotations #662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngoiyaeric wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Extend resolution search labels and GeoJSON annotations #662

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions app/actions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,9 @@ async function submit(formData?: FormData, skip?: boolean) {
},
properties: {
name: f.name,
description: f.description
description: f.description,
featureCategory: f.featureCategory,
displayLabel: f.displayLabel
}
}))
};
Expand Down Expand Up @@ -180,7 +182,9 @@ async function submit(formData?: FormData, skip?: boolean) {
}
return m
});
const relatedQueries = await querySuggestor(uiStream, sanitizedMessages);
const detectedFeatures = analysisResult.geoJson?.features?.map((f: any) => `${f.name} (${f.featureCategory || "other"})`).join(", ");

@coderabbitai coderabbitai Bot Jun 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | 💤 Low value

Consider stronger typing for feature mapping.

The code uses (f: any) which reduces type safety. Since analysisResult is of type ResolutionSearch (inferred from resolutionSearchSchema), the features array has a known structure.

♻️ Optional improvement for type safety 
-const detectedFeatures = analysisResult.geoJson?.features?.map((f: any) => `${f.name} (${f.featureCategory || "other"})`).join(", ");
+const detectedFeatures = analysisResult.geoJson?.features?.map((f) => `${f.name} (${f.featureCategory || "other"})`).join(", ");

TypeScript will infer the correct type from the schema, providing better autocomplete and type checking.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const detectedFeatures = analysisResult.geoJson?.features?.map((f: any) => `${f.name} (${f.featureCategory || "other"})`).join(", ");
const detectedFeatures = analysisResult.geoJson?.features?.map((f) => `${f.name} (${f.featureCategory || "other"})`).join(", ");
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` at line 185, Replace the use of the unsafe any in the
detectedFeatures mapping by typing the feature parameter with the schema-derived
type; e.g., import or reference the ResolutionSearch type (from
resolutionSearchSchema) and change (f: any) to (f:
ResolutionSearch["geoJson"]["features"][number]) or create a local alias like
type Feature = NonNullable<ResolutionSearch["geoJson"]>["features"][number] and
use (f: Feature) so TypeScript infers properties like name and featureCategory
safely when computing detectedFeatures.

const detectedFeaturesSummary = detectedFeatures ? `Detected: ${detectedFeatures}` : undefined;
const relatedQueries = await querySuggestor(uiStream, sanitizedMessages, detectedFeaturesSummary);
uiStream.append(
<Section title="Follow-up">
<FollowupPanel />
Expand All @@ -207,7 +211,10 @@ async function submit(formData?: FormData, skip?: boolean) {
geoJson: geoJson, // Use reconstructed GeoJSON for storage/UI
image: dataUrl,
mapboxImage: mapboxDataUrl,
googleImage: googleDataUrl
googleImage: googleDataUrl,
mapboxImageLabel: analysisResult.mapboxImageLabel,
googleImageLabel: analysisResult.googleImageLabel,
analysisFocus: analysisResult.analysisFocus
}),
type: 'resolution_search_result'
},
Expand Down Expand Up @@ -242,6 +249,9 @@ async function submit(formData?: FormData, skip?: boolean) {
mapboxImage={mapboxDataUrl || undefined}
googleImage={googleDataUrl || undefined}
initialImage={dataUrl}
mapboxImageLabel={undefined}
googleImageLabel={undefined}
analysisFocus={undefined}
/>
<BotMessage content={summaryStream.value} />
</Section>
Expand Down Expand Up @@ -836,6 +846,9 @@ export const getUIStateFromAIState = (aiState: AIState): UIState => {
const image = analysisResult.image as string;
const mapboxImage = analysisResult.mapboxImage as string;
const googleImage = analysisResult.googleImage as string;
const mapboxImageLabel = analysisResult.mapboxImageLabel as string;
const googleImageLabel = analysisResult.googleImageLabel as string;
const analysisFocus = analysisResult.analysisFocus as string;

return {
id,
Expand All @@ -845,6 +858,9 @@ export const getUIStateFromAIState = (aiState: AIState): UIState => {
mapboxImage={mapboxImage}
googleImage={googleImage}
initialImage={image}
mapboxImageLabel={mapboxImageLabel}
googleImageLabel={googleImageLabel}
analysisFocus={analysisFocus}
/>
{geoJson && (
<GeoJsonLayer id={id} data={geoJson} />
Expand Down
12 changes: 7 additions & 5 deletions components/compare-slider.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ interface CompareSliderProps {
leftImage: string
rightImage: string
className?: string
leftLabel?: string
rightLabel?: string
}

export function CompareSlider({ leftImage, rightImage, className }: CompareSliderProps) {
export function CompareSlider({ leftImage, rightImage, className, leftLabel, rightLabel }: CompareSliderProps) {
const [sliderPosition, setSliderPosition] = useState(50)
const containerRef = useRef<HTMLDivElement>(null)
const [containerWidth, setContainerWidth] = useState(0)
Expand Down Expand Up @@ -80,11 +82,11 @@ export function CompareSlider({ leftImage, rightImage, className }: CompareSlide
</div>

{/* Labels */}
<div className="absolute bottom-2 left-2 px-2 py-1 bg-black/50 text-white text-[10px] rounded pointer-events-none">
MAPBOX
<div className="absolute bottom-2 left-2 px-2 py-1 bg-black/50 text-white text-[10px] rounded pointer-events-none uppercase">
{leftLabel || 'MAPBOX'}
</div>
<div className="absolute bottom-2 right-2 px-2 py-1 bg-black/50 text-white text-[10px] rounded pointer-events-none">
GOOGLE SATELLITE
<div className="absolute bottom-2 right-2 px-2 py-1 bg-black/50 text-white text-[10px] rounded pointer-events-none uppercase">
{rightLabel || 'GOOGLE SATELLITE'}
</div>
</div>
)
Expand Down
29 changes: 24 additions & 5 deletions components/resolution-carousel.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,19 @@ interface ResolutionCarouselProps {
mapboxImage?: string | null
googleImage?: string | null
initialImage?: string | null
mapboxImageLabel?: string
googleImageLabel?: string
analysisFocus?: string
}

export function ResolutionCarousel({ mapboxImage, googleImage, initialImage }: ResolutionCarouselProps) {
export function ResolutionCarousel({
mapboxImage,
googleImage,
initialImage,
mapboxImageLabel,
googleImageLabel,
analysisFocus
}: ResolutionCarouselProps) {
const actions = useActions<typeof AI>() as any
const [, setMessages] = useUIState<typeof AI>()
const [isAnalyzing, setIsAnalyzing] = React.useState(false)
Expand Down Expand Up @@ -75,12 +85,12 @@ export function ResolutionCarousel({ mapboxImage, googleImage, initialImage }: R
}

// Individual slides
if (mapboxImage) slides.push({ type: 'image', src: mapboxImage, showAnalysis: false, label: 'MAPBOX' })
if (googleImage) slides.push({ type: 'image', src: googleImage, showAnalysis: true, label: 'GOOGLE SATELLITE' })
if (mapboxImage) slides.push({ type: 'image', src: mapboxImage, showAnalysis: false, label: mapboxImageLabel || 'MAPBOX' })
if (googleImage) slides.push({ type: 'image', src: googleImage, showAnalysis: true, label: googleImageLabel || 'GOOGLE SATELLITE' })

// Fallback
if (slides.length === 0 && initialImage) {
slides.push({ type: 'image', src: initialImage, showAnalysis: false, label: 'MAP CAPTURE' })
slides.push({ type: 'image', src: initialImage, showAnalysis: false, label: analysisFocus || 'MAP CAPTURE' })
}

if (slides.length === 0) return null
Expand All @@ -102,6 +112,9 @@ export function ResolutionCarousel({ mapboxImage, googleImage, initialImage }: R
{isAnalyzing ? 'ANALYZING...' : 'QCX-TERRA ANALYSIS'}
</Button>
)}
<div className="mt-1 text-[10px] text-muted-foreground uppercase tracking-widest text-center">
{item.label}
</div>
</div>
)
}
Expand All @@ -115,7 +128,13 @@ export function ResolutionCarousel({ mapboxImage, googleImage, initialImage }: R
<CarouselItem key={index}>
<div className="flex flex-col items-center p-1">
{slide.type === 'compare' ? (
<CompareSlider leftImage={slide.left} rightImage={slide.right} className="w-full" />
<CompareSlider
leftImage={slide.left}
rightImage={slide.right}
className="w-full"
leftLabel={mapboxImageLabel}
rightLabel={googleImageLabel}
/>
) : (
<>
<ResolutionImage src={slide.src} className="mb-0 mt-0 w-full" />
Expand Down
24 changes: 15 additions & 9 deletions lib/agents/query-suggestor.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,27 @@ interface CacheEntry {
const queryCache = new Map<string, CacheEntry>();
const CACHE_TTL = 5 * 60 * 1000; // 5 minutes

function getCacheKey(messages: CoreMessage[]): string {
// Create a simple hash of the last few messages to use as cache key
function getCacheKey(messages: CoreMessage[], detectedFeatures?: string): string {
// Create a simple hash of the last few messages and detected features to use as cache key
const recentMessages = messages.slice(-3);
return JSON.stringify(recentMessages.map(m => ({
role: m.role,
content: typeof m.content === 'string' ? m.content : '[complex content]'
})));
return JSON.stringify({
messages: recentMessages.map(m => ({
role: m.role,
content: typeof m.content === 'string' ? m.content : '[complex content]'
})),
detectedFeatures
});
}

export async function querySuggestor(
uiStream: ReturnType<typeof createStreamableUI>,
messages: CoreMessage[]
messages: CoreMessage[],
detectedFeatures?: string
) {
const objectStream = createStreamableValue<PartialRelated>()

// OPTIMIZATION: Check cache first
const cacheKey = getCacheKey(messages);
const cacheKey = getCacheKey(messages, detectedFeatures);
const cachedEntry = queryCache.get(cacheKey) as CacheEntry | undefined;

if (cachedEntry && Date.now() - cachedEntry.timestamp < CACHE_TTL) {
Expand All @@ -54,10 +58,12 @@ export async function querySuggestor(

let finalRelatedQueries: PartialRelated = {}

const systemPrompt = `Generate 3 follow-up queries that explore the subject matter deeper. Format as JSON with an "items" array containing objects with "query" fields. Keep queries concise and relevant.${detectedFeatures ? `\n\nContext - Detected Features: ${detectedFeatures}. Incorporate these features into the suggested queries (e.g., "Tell me more about [Feature Name]" or "What is the significance of [Feature Name]?").` : ''}`;

// OPTIMIZATION: Use a more concise system prompt to reduce token usage
const result = await streamObject({
model: (await getModel()) as LanguageModel,
system: `Generate 3 follow-up queries that explore the subject matter deeper. Format as JSON with an "items" array containing objects with "query" fields. Keep queries concise and relevant.`,
system: systemPrompt,
messages,
schema: relatedSchema,
temperature: 0.7, // Lower temperature for more consistent results
Expand Down
4 changes: 4 additions & 0 deletions lib/agents/resolution-search.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,13 @@ Use these user-drawn areas/lines as primary areas of interest for your analysis.
5. **COG Applicability:** Determine if this location would benefit from Cloud Optimized GeoTIFF (COG) analysis for high-precision temporal or spectral data.
6. **News Integration:** Reference any recent news or events that may be relevant to the current state of the location.
7. **Structured Output:** Return your findings in a structured JSON format including summary, geoJson (if any), news context, and any extracted coordinates or COG information. Use the provided schema.
8. **Contextual Labeling:** Produce location-specific and feature-aware labels for the images (\`mapboxImageLabel\`, \`googleImageLabel\`, \`analysisFocus\`). Reference user-drawn features in these labels when they are present. Labels should be concise, uppercase-friendly, and descriptive of the analysis focus. Examples: "Analysis of drawn area: [feature type]", "[Location name] satellite view".
9. **Semantic Annotations:** When creating \`geoJson\` features, use Point features for POIs and landmarks, and Polygon features for land areas or regions of interest. Each feature must have a \`name\`, \`featureCategory\`, and \`displayLabel\` matching findings in your summary. Example: Create a Point for identified bridges with name 'Main Street Bridge', category 'infrastructure'.

Comment on lines +131 to 133

@coderabbitai coderabbitai Bot Jun 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align prompt instruction with schema optionality.

The prompt states that each feature "must have" featureCategory and displayLabel, but the schema defines these fields as optional (lines 23-26 in lib/schema/resolution-search.ts). This mismatch could lead to inconsistent model behavior or confusion about validation requirements.

🔧 Recommended alignment options

Option 1 (recommended): Update the prompt to reflect optionality:

-9. **Semantic Annotations:** When creating \`geoJson\` features, use Point features for POIs and landmarks, and Polygon features for land areas or regions of interest. Each feature must have a \`name\`, \`featureCategory\`, and \`displayLabel\` matching findings in your summary. Example: Create a Point for identified bridges with name 'Main Street Bridge', category 'infrastructure'.
+9. **Semantic Annotations:** When creating \`geoJson\` features, use Point features for POIs and landmarks, and Polygon features for land areas or regions of interest. Each feature must have a \`name\` and should include \`featureCategory\` and \`displayLabel\` when applicable to match findings in your summary. Example: Create a Point for identified bridges with name 'Main Street Bridge', category 'infrastructure', and displayLabel 'Main St. Bridge'.

Option 2: Update the schema to make fields required:

// In lib/schema/resolution-search.ts
featureCategory: z.enum(['poi', 'land_feature', 'infrastructure', 'drawn_area', 'other'])
  .describe('The category of the feature...'),
displayLabel: z.string().describe('A short label for map display or tooltips')

(Note: Option 1 is safer since downstream code already handles these as optional with fallbacks.)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/agents/resolution-search.tsx` around lines 131 - 133, Summary: The prompt
requires every geoJson feature to "must have" featureCategory and displayLabel
but the schema (featureCategory, displayLabel) marks them optional, causing a
mismatch. Fix: choose Option 1 (recommended) — update the prompt text in the
resolution-search prompt generator to state that featureCategory and
displayLabel are optional and should be provided when available (or include
sensible fallbacks), and ensure the prompt references mapboxImageLabel,
googleImageLabel, and analysisFocus accurately; alternatively, if you prefer
Option 2, change the zod schema definitions for featureCategory and displayLabel
in the resolution-search schema to required enums/strings (replace optional()
with required definitions) so they match the prompt; finally, verify any code
that serializes geoJson respects the chosen behavior (use Point vs Polygon and
include name, featureCategory, displayLabel when present).

Your analysis should be based on the visual information in the image, the temporal context provided, and your general knowledge. Do not attempt to access external websites or perform web searches beyond what has been provided.

In your summary, reference annotated features by name (e.g., "I've marked the identified commercial district as a polygon", "The red marker indicates the location of...").

Analyze the user's prompt and the image to provide a holistic understanding of the location with full temporal and contextual awareness.
`;

Expand Down
10 changes: 9 additions & 1 deletion lib/schema/resolution-search.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,22 @@ export const resolutionSearchSchema = z.object({
.or(z.array(z.array(z.array(z.number()))))
.describe('Coordinates for the geometry'),
name: z.string().describe('Name of the feature or point of interest'),
description: z.string().optional().describe('Description of the feature')
description: z.string().optional().describe('Description of the feature'),
featureCategory: z.enum(['poi', 'land_feature', 'infrastructure', 'drawn_area', 'other'])
.optional()
.describe('The category of the feature. Use poi for landmarks, land_feature for natural elements, infrastructure for man-made structures, and drawn_area for user-defined regions.'),
displayLabel: z.string().optional().describe('A short label for map display or tooltips')
}))
}).optional().describe('A collection of points of interest and classified land features to be overlaid on the map.'),

// Flattened top-level fields for better xAI compatibility
extractedLatitude: z.number().optional().describe('The extracted latitude of the center of the image.'),
extractedLongitude: z.number().optional().describe('The extracted longitude of the center of the image.'),

mapboxImageLabel: z.string().optional().describe('A location-specific label for the Mapbox image (e.g., "DOWNTOWN SAN FRANCISCO").'),
googleImageLabel: z.string().optional().describe('A location-specific label for the Google Satellite image (e.g., "SATELLITE VIEW - MISSION DISTRICT").'),
analysisFocus: z.string().optional().describe('A description of the overall analysis focus or user-drawn feature (e.g., "ANALYSIS OF COASTAL EROSION").'),

cogApplicable: z.boolean().optional().describe('Whether Cloud Optimized GeoTIFF (COG) data is applicable for this area.'),
cogDescription: z.string().optional().describe('Description of COG data availability or benefits.'),

Expand Down