goscraper

Goscraper is a backend that fetches and scrapes data from SRM Academia and parses it dynamically. The API is designed for performance, speed, and accuracy, accommodating changes in the HTML structure.

Important

This is part of a monorepo containing both frontend and backend components. Please fork the entire monorepo instead of cloning this backend separately.

Prerequisites

Before starting the application, you need to set up Supabase and configure the required environment variables.

1. Supabase Setup

First, create a new Supabase project and set up the required database tables:

Create the goscrape table:

create table public.goscrape (
  "regNumber" text not null,
  "user" text null,
  timetable text null,
  courses text null,
  attendance text null,
  marks text null,
  "lastUpdated" numeric null,
  token text not null,
  ophour text null default ''::text,
  constraint goscrape_pkey primary key ("regNumber", token),
  constraint goscrape_regNumber_key unique ("regNumber")
);

Create the gocal table:

create table public.gocal (
  id bigint generated by default as identity not null,
  date text null,
  month text null,
  day text null,
  "order" text null,
  event text null,
  created_at numeric null,
  constraint gocal_month_date_key unique ("month", "date")
);

The unique constraint is what makes writes idempotent. /calendar only writes when gocal is empty and does so from a goroutine, so two requests arriving before either finishes both see an empty table and both insert the whole calendar. SetEvent upserts on (month, date), which needs this constraint to resolve against.

Set up CRON Jobs

Warning

Install CRON Integration in Supabase first

Deleting old user data:

UPDATE goscrape
SET "user" = NULL, timetable = NULL, attendance = NULL, marks = NULL, courses = NULL
WHERE "lastUpdated" < (EXTRACT(EPOCH FROM NOW()) * 1000) - (12 * 60 * 60 * 1000);

Deleting old calendar:

CREATE OR REPLACE FUNCTION delete_from_gocal()
RETURNS void AS $$
BEGIN
    DELETE FROM gocal;
END;
$$ LANGUAGE plpgsql SECURITY INVOKER;

select
  cron.schedule (
    '0 0 * * *',
    'SELECT delete_from_gocal()'
  );

Important

This job is the only thing that refreshes the calendar. /calendar rescrapes solely when gocal is empty, so without it an instance keeps serving the planner it first fetched and never picks up a new semester.

2. Environment Variables

Environment variables are handled by the monorepo. Please refer to the main repository's README.md for complete setup instructions.

---

WHEN DEVELOPING LOCALLY

globals/DevMode defaults to false, which is what deployments need: while it is true the Authorization middleware in main.go is skipped entirely and every route answers without a token. Set it to true while working locally, and keep it out of your commits.

---

Setup Instructions

1. Fork the monorepo:

   git clone --recurse-submodules https://github.com/rahuletto/classpro
   cd classpro/backend
   

2. Install dependencies:

   go mod tidy
   

3. Development Run the application: (DEV SERVER)

   go run main.go
   

4. Build and Run the application: (BUILD SERVER)

   go build main.go
   ./main
   

Docker Setup

docker compose up --build

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any enhancements or bug fixes.