CSV Upload

.eslintrc.js

@@ -20,7 +20,7 @@ module.exports = {
   ],
   "rules": {
     "indent": [
-      "error",
+      "off",
       2
     ],
     "no-unused-vars": [

README.md

@@ -11,182 +11,7 @@ We will test the architecture by crawling all 100,000 or so U.S. school websites
 
 Downstream features on our bucket list include real-time metrics and access to scraped data, error checks and backup scrapers (including the simple wget algorithm), and toggles for capturing data over time with the Internet Archive. 
 
+## Running scraping server
 
-## Running the scraping server (Ubuntu)
-This requires a Redis server to handle tasks. The instructions below walk you through installing Redis, and [you can find more instructions here](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04) (Ubuntu 18.04). If not yet installed, you will also need to [install MongoDB](https://docs.mongodb.com/manual/installation/) (instructions below don't include this part). You will also need access to authorization credentials for Google Sign-In; [here are instructions to create these if you haven't used them before](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials). 
+The instructions to run the scraping server on Linux, Windows, and Mac are in the directory setup_guides.
 
-You will need 3 terminal windows for this, although [there are ways to run Redis and/or Flask headless to remove this need](https://askubuntu.com/questions/106351/running-programs-in-the-background-from-terminal)--the easiest being to add an ampersand (`&`) after the command. If you choose to have separate terminal windows (best for monitoring purposes), create a window for Redis, Flask, and React.
-
-### 1. Setup and start Redis on machine:
-```bash
-sudo apt-get install redis-server
-sudo nano /etc/redis/redis.conf # change 'supervised no' to 'supervised systemd'
-sudo systemctl restart redis.service
-sudo systemctl status redis # see if redis is actively running
-sudo nano /etc/redis/redis.conf 
-# uncomment the line: # bind 127.0.0.1 ::1 
-# Then restart Redis again if you made the previous change
-sudo systemctl restart redis
-```
-
-
-
-### 2. Install required packages and setup 
-Follow each of these steps from your *home directory* (which for our VMs this is `/vol_b/data/`).
-
-#### 2A. Create python 3 environment and install packages
-```bash
-python3 -m venv .venv # create specific crawling environment with packages we want; feel free to use an env name other than `.venv`
-source .venv/bin/activate # activate environment
-sudo apt update # get latest version info
-pip3 install -r requirements.txt
-npm --prefix ./client install
-```
-
-#### 2B. Set up MongoDB container
-When setting up MongoDB, for security we recommend using a custom username and password and forwarding to a different port like 27000. Bypassing these measures makes it likely that you will experience web hacks and attempts to blackmail you by compromising your data stored in Mongo. The code below offers a template for such security through several extensions on the basic command of `docker run mongo`. Be sure to also update [the scrapy settings.py file with your custom Mongo username and password](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L138-L140).
-```bash
-mkdir mongodata
-docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v /vol_b/data/mongodata:/data/db mongo
-```
-Description of relevant parameters, please update as appropriate:
-- `-e MONGO_INITDB_ROOT_USERNAME=admin`: Set Mongo username to `admin`
-- `-e MONGO_INITDB_ROOT_PASSWORD=mdipass`: Set Mongo password to `mdipass`
-- `-p 27000:27017`: adjust MongoDB port to 27000
-- `--log-opt max-size=500m`: Set maximum Mongo log size to 500 MB, to prevent overloading root drive
-- `-v /vol_b/data/mongodata:/data/db`: Set custom volume for Mongo output; update `/vol_b/data/` with your own target drive
-
-#### 2C. Set up user authorization with Google Sign-In
-Replace the Client ID in [`client/src/server-config.js`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/client/src/server-config.js#L5) and [`client/src/settings.py`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L150) with your own ([how to create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials)). You can enable crawling requests from your IP addresses (if not `localhost`) as "Authorized Javascript Origins" with your Client ID on [the Google Console Credentials page](https://console.developers.google.com/apis/credentials). The current repo uses the Client ID created by [Jaren Haber, PhD](https://www.jarenhaber.com/), which will work for the purposes of testing and developing the crawling server. 
-
-
-### 3. Run server
-Create three terminal screens: one for Redis, one for Flask, one for React. From each window: 
-- navigate to your home directory (in our VMs this is `/vol_b/data/`)
-- activate the python environment you set up in 2A above (default `source .venv/bin/activate`)
-- run one task per window as follows.
-
-##### 3A. In Redis window (must be in venv):
-```bash
-cd crawler
-rq worker crawling-tasks --path . # run Redis
-```
-
-#### 3B. In Flask window (must be in venv): 
-```bash
-export CLIENT_ORIGIN=http://localhost:3000
-export MONGO_URI=mongodb://localhost:27000
-export SERVER_PORT=5000
-cd crawler/crawler
-python app.py # run Flask
-```
-The environment variables guide the flask server. The values shown are the default values.
- - `CLIENT_ORIGIN` is the client it should accept requests from
- - `MONGO_URI` is where it should send database requests
- - `SERVER_PORT` is what port should the server run on
-
-### 3C. In React window:
-```bash
-export REACT_APP_SERVER_URL=http://localhost:5000
-cd client
-npm start # run React server
-```
-The environment variable here, `REACT_APP_SERVER_URL`, is the address of the flask server, to which the React client should send server url requests.
-
-
-### 4. Navigate the client from your web browser at `http://localhost:3000/`
-This will open up the home page.
-
-## Running the scraping server (macOS)
-
-### 1. Homebrew and Conda
-First, make sure you have both [Homebrew](https://brew.sh)
-```bash
-brew -v
-```
-and [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/macos.html#install-macos-silent) installed.
-```bash
-conda -V
-```
-
-### 2. Redis
-To install Redis on Mac
-```bash
-brew install redis
-```
-and start the redis server using _brew service._
-```bash
-brew services start redis
-```
-
-### 3. Conda Environment
-Create a Conda environment named `wc-server` using Python 3.10
-```bash
-conda create --name wc-server python=3.10
-```
-Activate the environment
-```bash
-conda activate wc-server
-```
-Install dependencies using `pip`
-```bash
-pip install -r requirements.txt
-```
-Setup environment variables
-```bash
-echo "CLIENT_ORIGIN=http://localhost:3000 MONGO_URI=mongodb://localhost:27000 SERVER_PORT=5000 REACT_APP_SERVER_URL=http://localhost:5000" | xargs conda env config vars set
-
-conda deactivate
-conda activate wc-server
-```
-
-### 4. Docker
-To install the Docker macOS App
-```bash
-brew install --cask docker
-brew install docker-machine
-```
-In the project folder, create a folder called `mongodata`, we will use it as the mounting point of our docker container.
-```bash
-mkdir mongodata
-```
-Then we can spawn the docker instance using
-```bash
-docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v $PWD/mongodata:/data/db mongo
-```
-
-### 5. Node modules
-To install Node.js and NPM
-```bash
-brew install node
-```
-and install all the necessary node modules (such as React)
-```bash
-npm --prefix ./client install
-```
-
-### 6. Set up user authorization with Google Sign-In
-Replace the Client ID in [`client/src/server-config.js`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/client/src/server-config.js#L5) and [`client/src/settings.py`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L150) with your own ([how to create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials)). You can enable crawling requests from your IP addresses (if not `localhost`) as "Authorized Javascript Origins" with your Client ID on [the Google Console Credentials page](https://console.developers.google.com/apis/credentials). The current repo uses the Client ID created by [Jaren Haber, PhD](https://www.jarenhaber.com/), which will work for the purposes of testing and developing the crawling server.
-
-### 7. Run Redis Queue, Flask, React Server
-In three Terminal windows, navigate to the project folder and run the following commands.
-
-Terminal 1:
-```bash
-conda activate wc-server
-rq worker crawling-tasks --path ./crawler
-```
-
-Terminal 2:
-```bash
-conda activate wc-server
-python ./crawler/crawler/app.py
-```
-
-Terminal 3:
-```bash
-conda activate wc-server
-npm --prefix ./client start
-```
-
-At this point, you should be able to see the web interface running on [localhost:3000](http://localhost:3000)

client/src/App.js

@@ -5,6 +5,8 @@ import NewJob from "./pages/NewJob.js";
 import Job from "./pages/Job.js";
 import Home from "./pages/Home.js";
 import ResponsiveAppBar from "./components/Navbar";
+import theme from "./styles/Theme";
+import {ThemeProvider} from "@mui/system";
 
 // import './App.css';
 
@@ -20,14 +22,16 @@ class App extends Component {
 
   render() {
     return (
-      <BrowserRouter>
-        <Routes>
-          <Route exact path="/" element={<Home />} />
-          <Route exact path="/new-job" element={<NewJob />} />
-          <Route exact path="/jobs" element={<JobsDashboard />} />
-          <Route exact path="/job/:id" element={<Job />} />
-        </Routes>
-      </BrowserRouter>
+      <ThemeProvider theme={theme}>
+        <BrowserRouter>
+          <Routes>
+            <Route exact path="/" element={<Home />} />
+            <Route exact path="/new-job" element={<NewJob />} />
+            <Route exact path="/jobs" element={<JobsDashboard />} />
+            <Route exact path="/job/:id" element={<Job />} />
+          </Routes>
+        </BrowserRouter>
+      </ThemeProvider>
     );
   }
 }

client/src/components/SingleJob.js

@@ -1,9 +1,8 @@
 import {
-  Card, CardContent, CardHeader, Typography,
+  Card, CardContent, CardHeader, Typography, Button,
   CardActions, List, ListItem, ListItemButton, ListItemText
 } from "@mui/material";
 import React, { Component } from "react";
-import {TopButton} from "../styles/JobsStyled";
 import {fetchWithUserToken} from "../util/AuthManager";
 
 class SingleJob extends Component {
@@ -62,23 +61,20 @@ class SingleJob extends Component {
         {
           {
             "Ongoing": <div><CardContent><p>Process is Running</p></CardContent>
-              <CardActions><TopButton
-                variant="extended"
-                color="primary"
-                onClick={this.killFunc}>
-                                    Kill
-              </TopButton></CardActions>
+              <CardActions>
+                <Button variant="contained" color="primary" onClick={this.killFunc}>
+                  Kill
+                </Button>
+              </CardActions>
             </div>,
 
             "Error": <CardContent><p>Process Errored</p></CardContent>,
             "Finished": <div><CardContent><p>Process Completed</p></CardContent>
-              <CardActions><TopButton
-                variant="extended"
-                color="primary"
-                href={`/api/jobs/${this.props.id}/download`}
-              >
-                                Download
-              </TopButton></CardActions>
+              <CardActions>
+                <Button variant="contained" color="primary" href={`/api/jobs/${this.props.id}/download`}>
+                  Download
+                </Button>
+              </CardActions>
             </div>,
             "Cancelled": <CardContent><p>Process Cancelled</p></CardContent>,
             "Failed": <CardContent><p>Process Failed</p></CardContent>

client/src/pages/JobsDashboard.js

@@ -4,14 +4,15 @@ import {Copyright} from "../components/Copyright";
 import ResponsiveAppBar from "../components/Navbar";
 import {getComparator, stableSort} from "../util/jobSortingHelpers";
 import {
-  Grid, Table, TableBody, Container,
+  Grid, Table, TableBody, Container, Button,
   TableCell, TableHead, TablePagination,
   TableRow, Toolbar, Box, TableSortLabel,
 } from "@mui/material";
 import AddIcon from "@mui/icons-material/Add";
 import { useNavigate } from "react-router-dom";
 import {WCTableContainer, WCTablePaper} from "../styles/DatasetsStyled";
-import {JobTableToolBarTitle, TopButton, RootDiv, Main} from "../styles/JobsStyled";
+import {JobTableToolBarTitle, RootDiv, Main} from "../styles/JobsStyled";
+import {TopButtonsGrid} from "../styles/DashboardStyled";
 
 const jobsTableHeader = [
   {id: "title", label: "Title", minWidth: 120, align: "left"},
@@ -22,27 +23,20 @@ const jobsTableHeader = [
   {id: "more", label: " ", minWidth: 40, align: "left"}
 ];
 
+
+
 function TopButtons(props) {
 
   const handleNewJobClick = () => {
     props.navigate("/new-job");
   };
 
   return (
-    <Grid container item
-      direction="row"
-      justify="flex-start"
-      alignItems="center"
-    >
-      <TopButton
-        variant="extended"
-        color="primary"
-        onClick={() => handleNewJobClick()}
-      >
-        <AddIcon />
-                NEW JOB
-      </TopButton>
-    </Grid>
+    <TopButtonsGrid container item direction="row" justify="flex-start" alignItems="center">
+      <Button variant="contained" color="primary" startIcon={<AddIcon />} onClick={() => handleNewJobClick()}>
+        New Job
+      </Button>
+    </TopButtonsGrid>
   );
 }