Skip to content

Improve performance of daily mongodb loads #6

@Ameya05

Description

@Ameya05

The current mongodb implementation does the following things daily -

  1. Uses the existing SQLite intermediate databases for all stations.
  2. Drops existing mongodb database.
  3. Computes data states for all stations and all dates (from 01/01/1994 to today).
  4. Inserts these values into a new mongodb database.

The current process takes upwards of 24 hours to process approximately upto data states 10,000 GPS Stations. Daily re-computation of Historical data states is redundant and can be avoided.

A simple performance improvement would be compute only the end_date's data state and attach it to the mongodb collection everyday, instead of dropping and creating from scratch.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions