diff --git a/.eslintrc.js b/.eslintrc.js index 9749cb0..2d5dbff 100644 --- a/.eslintrc.js +++ b/.eslintrc.js @@ -20,7 +20,7 @@ module.exports = { ], "rules": { "indent": [ - "error", + "off", 2 ], "no-unused-vars": [ diff --git a/README.md b/README.md index b27081e..d417257 100644 --- a/README.md +++ b/README.md @@ -11,182 +11,7 @@ We will test the architecture by crawling all 100,000 or so U.S. school websites Downstream features on our bucket list include real-time metrics and access to scraped data, error checks and backup scrapers (including the simple wget algorithm), and toggles for capturing data over time with the Internet Archive. +## Running scraping server -## Running the scraping server (Ubuntu) -This requires a Redis server to handle tasks. The instructions below walk you through installing Redis, and [you can find more instructions here](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04) (Ubuntu 18.04). If not yet installed, you will also need to [install MongoDB](https://docs.mongodb.com/manual/installation/) (instructions below don't include this part). You will also need access to authorization credentials for Google Sign-In; [here are instructions to create these if you haven't used them before](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials). +The instructions to run the scraping server on Linux, Windows, and Mac are in the directory setup_guides. -You will need 3 terminal windows for this, although [there are ways to run Redis and/or Flask headless to remove this need](https://askubuntu.com/questions/106351/running-programs-in-the-background-from-terminal)--the easiest being to add an ampersand (`&`) after the command. If you choose to have separate terminal windows (best for monitoring purposes), create a window for Redis, Flask, and React. - -### 1. Setup and start Redis on machine: -```bash -sudo apt-get install redis-server -sudo nano /etc/redis/redis.conf # change 'supervised no' to 'supervised systemd' -sudo systemctl restart redis.service -sudo systemctl status redis # see if redis is actively running -sudo nano /etc/redis/redis.conf -# uncomment the line: # bind 127.0.0.1 ::1 -# Then restart Redis again if you made the previous change -sudo systemctl restart redis -``` - - - -### 2. Install required packages and setup -Follow each of these steps from your *home directory* (which for our VMs this is `/vol_b/data/`). - -#### 2A. Create python 3 environment and install packages -```bash -python3 -m venv .venv # create specific crawling environment with packages we want; feel free to use an env name other than `.venv` -source .venv/bin/activate # activate environment -sudo apt update # get latest version info -pip3 install -r requirements.txt -npm --prefix ./client install -``` - -#### 2B. Set up MongoDB container -When setting up MongoDB, for security we recommend using a custom username and password and forwarding to a different port like 27000. Bypassing these measures makes it likely that you will experience web hacks and attempts to blackmail you by compromising your data stored in Mongo. The code below offers a template for such security through several extensions on the basic command of `docker run mongo`. Be sure to also update [the scrapy settings.py file with your custom Mongo username and password](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L138-L140). -```bash -mkdir mongodata -docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v /vol_b/data/mongodata:/data/db mongo -``` -Description of relevant parameters, please update as appropriate: -- `-e MONGO_INITDB_ROOT_USERNAME=admin`: Set Mongo username to `admin` -- `-e MONGO_INITDB_ROOT_PASSWORD=mdipass`: Set Mongo password to `mdipass` -- `-p 27000:27017`: adjust MongoDB port to 27000 -- `--log-opt max-size=500m`: Set maximum Mongo log size to 500 MB, to prevent overloading root drive -- `-v /vol_b/data/mongodata:/data/db`: Set custom volume for Mongo output; update `/vol_b/data/` with your own target drive - -#### 2C. Set up user authorization with Google Sign-In -Replace the Client ID in [`client/src/server-config.js`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/client/src/server-config.js#L5) and [`client/src/settings.py`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L150) with your own ([how to create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials)). You can enable crawling requests from your IP addresses (if not `localhost`) as "Authorized Javascript Origins" with your Client ID on [the Google Console Credentials page](https://console.developers.google.com/apis/credentials). The current repo uses the Client ID created by [Jaren Haber, PhD](https://www.jarenhaber.com/), which will work for the purposes of testing and developing the crawling server. - - -### 3. Run server -Create three terminal screens: one for Redis, one for Flask, one for React. From each window: -- navigate to your home directory (in our VMs this is `/vol_b/data/`) -- activate the python environment you set up in 2A above (default `source .venv/bin/activate`) -- run one task per window as follows. - -##### 3A. In Redis window (must be in venv): -```bash -cd crawler -rq worker crawling-tasks --path . # run Redis -``` - -#### 3B. In Flask window (must be in venv): -```bash -export CLIENT_ORIGIN=http://localhost:3000 -export MONGO_URI=mongodb://localhost:27000 -export SERVER_PORT=5000 -cd crawler/crawler -python app.py # run Flask -``` -The environment variables guide the flask server. The values shown are the default values. - - `CLIENT_ORIGIN` is the client it should accept requests from - - `MONGO_URI` is where it should send database requests - - `SERVER_PORT` is what port should the server run on - -### 3C. In React window: -```bash -export REACT_APP_SERVER_URL=http://localhost:5000 -cd client -npm start # run React server -``` -The environment variable here, `REACT_APP_SERVER_URL`, is the address of the flask server, to which the React client should send server url requests. - - -### 4. Navigate the client from your web browser at `http://localhost:3000/` -This will open up the home page. - -## Running the scraping server (macOS) - -### 1. Homebrew and Conda -First, make sure you have both [Homebrew](https://brew.sh) -```bash -brew -v -``` -and [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/macos.html#install-macos-silent) installed. -```bash -conda -V -``` - -### 2. Redis -To install Redis on Mac -```bash -brew install redis -``` -and start the redis server using _brew service._ -```bash -brew services start redis -``` - -### 3. Conda Environment -Create a Conda environment named `wc-server` using Python 3.10 -```bash -conda create --name wc-server python=3.10 -``` -Activate the environment -```bash -conda activate wc-server -``` -Install dependencies using `pip` -```bash -pip install -r requirements.txt -``` -Setup environment variables -```bash -echo "CLIENT_ORIGIN=http://localhost:3000 MONGO_URI=mongodb://localhost:27000 SERVER_PORT=5000 REACT_APP_SERVER_URL=http://localhost:5000" | xargs conda env config vars set - -conda deactivate -conda activate wc-server -``` - -### 4. Docker -To install the Docker macOS App -```bash -brew install --cask docker -brew install docker-machine -``` -In the project folder, create a folder called `mongodata`, we will use it as the mounting point of our docker container. -```bash -mkdir mongodata -``` -Then we can spawn the docker instance using -```bash -docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v $PWD/mongodata:/data/db mongo -``` - -### 5. Node modules -To install Node.js and NPM -```bash -brew install node -``` -and install all the necessary node modules (such as React) -```bash -npm --prefix ./client install -``` - -### 6. Set up user authorization with Google Sign-In -Replace the Client ID in [`client/src/server-config.js`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/client/src/server-config.js#L5) and [`client/src/settings.py`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L150) with your own ([how to create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials)). You can enable crawling requests from your IP addresses (if not `localhost`) as "Authorized Javascript Origins" with your Client ID on [the Google Console Credentials page](https://console.developers.google.com/apis/credentials). The current repo uses the Client ID created by [Jaren Haber, PhD](https://www.jarenhaber.com/), which will work for the purposes of testing and developing the crawling server. - -### 7. Run Redis Queue, Flask, React Server -In three Terminal windows, navigate to the project folder and run the following commands. - -Terminal 1: -```bash -conda activate wc-server -rq worker crawling-tasks --path ./crawler -``` - -Terminal 2: -```bash -conda activate wc-server -python ./crawler/crawler/app.py -``` - -Terminal 3: -```bash -conda activate wc-server -npm --prefix ./client start -``` - -At this point, you should be able to see the web interface running on [localhost:3000](http://localhost:3000) diff --git a/client/src/App.js b/client/src/App.js index 30b28fa..78bb2b2 100644 --- a/client/src/App.js +++ b/client/src/App.js @@ -5,6 +5,8 @@ import NewJob from "./pages/NewJob.js"; import Job from "./pages/Job.js"; import Home from "./pages/Home.js"; import ResponsiveAppBar from "./components/Navbar"; +import theme from "./styles/Theme"; +import {ThemeProvider} from "@mui/system"; // import './App.css'; @@ -20,14 +22,16 @@ class App extends Component { render() { return ( - - - } /> - } /> - } /> - } /> - - + + + + } /> + } /> + } /> + } /> + + + ); } } diff --git a/client/src/components/SingleJob.js b/client/src/components/SingleJob.js index 49a3d14..f0c528e 100644 --- a/client/src/components/SingleJob.js +++ b/client/src/components/SingleJob.js @@ -1,9 +1,8 @@ import { - Card, CardContent, CardHeader, Typography, + Card, CardContent, CardHeader, Typography, Button, CardActions, List, ListItem, ListItemButton, ListItemText } from "@mui/material"; import React, { Component } from "react"; -import {TopButton} from "../styles/JobsStyled"; import {fetchWithUserToken} from "../util/AuthManager"; class SingleJob extends Component { @@ -62,23 +61,20 @@ class SingleJob extends Component { { { "Ongoing":

Process is Running

 - - Kill - + + +
, "Error":

Process Errored

, "Finished":

Process Completed

 - - Download - + + +
, "Cancelled":

Process Cancelled

, "Failed":

Process Failed

 diff --git a/client/src/pages/JobsDashboard.js b/client/src/pages/JobsDashboard.js index a436894..e97ae03 100644 --- a/client/src/pages/JobsDashboard.js +++ b/client/src/pages/JobsDashboard.js @@ -4,14 +4,15 @@ import {Copyright} from "../components/Copyright"; import ResponsiveAppBar from "../components/Navbar"; import {getComparator, stableSort} from "../util/jobSortingHelpers"; import { - Grid, Table, TableBody, Container, + Grid, Table, TableBody, Container, Button, TableCell, TableHead, TablePagination, TableRow, Toolbar, Box, TableSortLabel, } from "@mui/material"; import AddIcon from "@mui/icons-material/Add"; import { useNavigate } from "react-router-dom"; import {WCTableContainer, WCTablePaper} from "../styles/DatasetsStyled"; -import {JobTableToolBarTitle, TopButton, RootDiv, Main} from "../styles/JobsStyled"; +import {JobTableToolBarTitle, RootDiv, Main} from "../styles/JobsStyled"; +import {TopButtonsGrid} from "../styles/DashboardStyled"; const jobsTableHeader = [ {id: "title", label: "Title", minWidth: 120, align: "left"}, @@ -22,6 +23,8 @@ const jobsTableHeader = [ {id: "more", label: " ", minWidth: 40, align: "left"} ]; + + function TopButtons(props) { const handleNewJobClick = () => { @@ -29,20 +32,11 @@ function TopButtons(props) { }; return ( - - handleNewJobClick()} - > - - NEW JOB - - + + + ); } diff --git a/client/src/pages/NewJob.js b/client/src/pages/NewJob.js index 086d99d..48af51a 100644 --- a/client/src/pages/NewJob.js +++ b/client/src/pages/NewJob.js @@ -6,16 +6,33 @@ import {Copyright} from "../components/Copyright"; // Material UI import { - TextField, Box, Grid, Typography, - CardContent, Container, Card, FormGroup, CardActions + TextField, Box, Grid, Button, + CardContent, Container, Card, FormGroup, CardActions, CardHeader } from "@mui/material"; import DeleteIcon from "@mui/icons-material/Delete"; - +import SendIcon from "@mui/icons-material/Send"; +import AddIcon from "@mui/icons-material/Add"; +import FileUploadIcon from "@mui/icons-material/FileUpload"; // Styles -import { RootDiv, Main, TopButton } from "../styles/JobsStyled"; +import { RootDiv, Main } from "../styles/JobsStyled"; +import { styled } from "@mui/material/styles"; import ResponsiveAppBar from "../components/Navbar"; import {useNavigate} from "react-router-dom"; +const Input = styled("input")` + display: none; +`; + +const SubmitButton = styled(Button)` + margin-left: auto; +`; + +const NewJobCard = styled(Card)` + width: 50%; + min-width: 400px; + margin: 30px auto; + padding: 10px 30px 30px; +`; class _NewJob extends Component { @@ -23,33 +40,51 @@ class _NewJob extends Component { super(props); this.state = { title: "", - urls: [""] + csv_file: null, + urls: [], }; } - sendURLs = (urls, title) => { - fetchWithUserToken("/api/jobs/create", { - method: "POST", - body: JSON.stringify({urls: urls, title: title}) - }) - .then(res => { - this.props.navigate(`/job/${res.job_id}`); - console.log(this.state); - }); - }; - - handleSubmit = (event) => { - console.log("A name was submitted: ", this.state); - if (this.state.urls.length === 1 && this.state.urls[0] === "") { + if (!this.state.title || (!this.state.urls.length && !this.state.csv_file)) { return; } event.preventDefault(); - this.sendURLs(this.state.urls, this.state.title); + const requestURI = "/api/jobs/create"; + let request; + if (this.state.csv_file) { + const formData = new FormData(); + formData.append("csv_file", this.state.csv_file); + formData.append("title", this.state.title); + request = fetchWithUserToken(requestURI, { + method: "POST", + body: formData + }); + } else { + request = fetchWithUserToken("/api/jobs/create", { + method: "POST", + body: JSON.stringify({urls: this.state.urls, title: this.state.title}) + }); + } + + request.then(res => { + this.props.navigate(`/job/${res.job_id}`); + console.log(this.state); + }); + }; + + handleFileUpload = (event) => { + this.setState({ + csv_file: event.target.files[0], + urls: [], + }); }; addURL = () => { - this.setState({urls: this.state.urls.concat([""])}); + this.setState({ + csv_file: null, + urls: this.state.urls.concat([""]), + }); }; removeURL = (i) => { @@ -67,84 +102,58 @@ class _NewJob extends Component { {/**/}
- - + + + - - Enter Title - - {this.setState({title: event.target.value});}} + { + this.setState({title: event.target.value}); + }} /> - - - Enter URLs - - {this.state.urls.map( (url, i) => { - return ( - - { - let urls = this.state.urls; - urls[i] = event.target.value; - this.setState({urls: urls}); - }}/> - - {i > 0 ? - - :

} - - );})} + { + this.state.urls.map( (url, i) => { + return ( + + { + let urls = this.state.urls; + urls[i] = event.target.value; + this.setState({urls: urls}); + }}/> + + );} + ) + } + { + this.state.csv_file && + + } - - - Add - - - Submit - + + + + }> + Submit + - + diff --git a/client/src/styles/DashboardStyled.js b/client/src/styles/DashboardStyled.js index 776b7f3..fc14502 100644 --- a/client/src/styles/DashboardStyled.js +++ b/client/src/styles/DashboardStyled.js @@ -1,6 +1,10 @@ import {styled} from "@mui/system"; import {Card, CardMedia, Grid, Paper} from "@mui/material"; +export const TopButtonsGrid = styled(Grid)` + margin: 20px 0 20px 0; +`; + export const StatCard = styled(Card)` max-width: 300px; max-height: 300px; diff --git a/client/src/styles/Theme.js b/client/src/styles/Theme.js new file mode 100644 index 0000000..b47dd44 --- /dev/null +++ b/client/src/styles/Theme.js @@ -0,0 +1,25 @@ +import { createTheme } from "@mui/material/styles"; +import { blue } from "@mui/material/colors"; + +const theme = createTheme({ + palette: { + primary: { + main: "#1976d2", + }, + secondary: { + main: "#A2D2FF", + light: "#BDE0FE", + }, + background: { + default: "#FDFFFC", + }, + }, + + typography: { + fontFamily: [ + "Nunito", "-apple-system", "sans-serif" + ].join(",") + } +}); + +export default theme; diff --git a/environment.yml b/environment.yml index 596bbc4..a0755b6 100644 --- a/environment.yml +++ b/environment.yml @@ -2,22 +2,22 @@ name: wc-server channels: - defaults dependencies: - - bzip2=1.0.8=h1de35cc_0 - - ca-certificates=2021.10.26=hecd8cb5_2 - - libcxx=12.0.0=h2f01273_0 - - libffi=3.3=hb1e8313_2 - - ncurses=6.3=hca72f7f_2 - - openssl=1.1.1m=hca72f7f_0 - - pip=21.2.4=py310hecd8cb5_0 - - python=3.10.0=hdfd78df_3 - - readline=8.1.2=hca72f7f_1 - - setuptools=58.0.4=py310hecd8cb5_0 - - sqlite=3.37.0=h707629a_0 - - tk=8.6.11=h7bc2e8c_0 - - tzdata=2021e=hda174b7_0 - - xz=5.2.5=h1de35cc_0 - - zlib=1.2.11=h4dc903c_4 + - pip + - python=3.8.5 - pip: + - libffi==3.3 + - ncurses==6.3 + - openssl==1.1.1m + - readline==8.1.2 + - setuptools==58.0.4 + - sqlite==3.37.0 + - tzdata==2021e + - xz==5.2.5 + - zlib==1.2.11 + - libcxx==12.0.0 + - ca-certificates==2021.10.26 + - tk==8.6.11 + - bzip2==1.0.8 - aniso8601==9.0.1 - anyio==3.5.0 - appnope==0.1.2 diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index f3c85ee..0000000 --- a/requirements.txt +++ /dev/null @@ -1,91 +0,0 @@ -aniso8601==9.0.1 -argcomplete==1.10.0 -attrs==20.3.0 -Authlib==0.15.5 -Automat==20.2.0 -beautifulsoup4==4.8.0 -billiard==3.6.3.0 -cachetools==5.0.0 -certifi==2020.12.5 -cffi==1.14.4 -chardet==3.0.4 -charset-normalizer==2.0.11 -click==8.0.3 -constantly==15.1.0 -cryptography==3.3.2 -cssselect==1.1.0 -Deprecated==1.2.13 -docx2txt==0.8 -EbookLib==0.17.1 -extract-msg==0.23.1 -filelock==3.0.12 -Flask==1.1.2 -Flask-RESTful==0.3.8 -google==3.0.0 -google-api-core==2.4.0 -google-api-python-client==2.33.0 -google-auth==2.6.0 -google-auth-httplib2==0.1.0 -googleapis-common-protos==1.54.0 -h2==3.2.0 -hpack==3.0.0 -html5lib==1.1 -httplib2==0.20.2 -hyperframe==5.2.0 -hyperlink==20.0.1 -idna==2.10 -IMAPClient==2.1.0 -incremental==17.5.0 -itemadapter==0.2.0 -itemloaders==1.0.4 -itsdangerous==2.0.1 -Jinja2==3.0.3 -jmespath==0.10.0 -lxml==4.6.5 -MarkupSafe==2.0.1 -olefile==0.46 -packaging==21.3 -parsel==1.6.0 -pdfminer.six==20181108 -Pillow==9.0.0 -priority==1.3.0 -Protego==0.1.16 -protobuf==3.19.4 -pyasn1==0.4.8 -pyasn1-modules==0.2.8 -pycparser==2.20 -pycryptodome==3.9.9 -PyDispatcher==2.0.5 -PyHamcrest==2.0.2 -pymongo==3.11.2 -pyOpenSSL==20.0.0 -pyparsing==3.0.7 -python-pptx==0.6.18 -pytz==2020.4 -queuelib==1.5.0 -redis==4.1.2 -regex==2020.11.13 -requests==2.25.0 -requests-file==1.5.1 -rq==1.8.1 -rsa==4.8 -Scrapy==2.5.1 -scrapyscript==1.1.0 -service-identity==18.1.0 -six==1.12.0 -sortedcontainers==2.3.0 -soupsieve==2.0.1 -SpeechRecognition==3.8.1 -textract==1.6.3 -tldextract==3.1.0 -Twisted==20.3.0 -tzlocal==1.5.1 -uritemplate==4.1.1 -urllib3==1.26.5 -w3lib==1.22.0 -webencodings==0.5.1 -Werkzeug==2.0.2 -wrapt==1.13.3 -xlrd==1.2.0 -XlsxWriter==1.3.7 -zope.interface==5.2.0 diff --git a/server/__init__.py b/server/__init__.py index 415c2d6..889d515 100644 --- a/server/__init__.py +++ b/server/__init__.py @@ -11,6 +11,8 @@ def create_app(test_config=None): if test_config is not None: app.config.from_mapping(test_config) + + app.config["UPLOAD_FOLDER"] = settings.UPLOAD_FOLDER from .home import views as home_views app.register_blueprint(home_views.bp) diff --git a/server/crawler/output.py b/server/crawler/output.py index 2a28124..4fab12a 100644 --- a/server/crawler/output.py +++ b/server/crawler/output.py @@ -26,12 +26,17 @@ def bucket_export(folder, bucket_name, job_id): def items_export(folder, job_id): - docs = db[settings.MONGO_COLLECTION_ITEMS]\ - .find({"job_id": job_id}) - if docs.count() == 0: + # if db[settings.MONGO_COLLECTION_ITEMS].count_documents({"job_id": job_id}) == 0: + # return + + collection = db[settings.MONGO_COLLECTION_ITEMS] + + if collection.count_documents({"job_id": job_id}) == 0: return + docs = collection.find({"job_id": job_id}) + fields = list(docs[0].keys()) output_file = os.path.join(folder, f"{job_id}.csv") with open(output_file, "w", newline='') as csvfile: diff --git a/server/crawler/run_spider.py b/server/crawler/run_spider.py index 9d78611..d16874b 100755 --- a/server/crawler/run_spider.py +++ b/server/crawler/run_spider.py @@ -8,6 +8,8 @@ NOTE: by default, data doesn’t persist when that container no longer exists. """ +import pandas as pd + from scrapy.utils.project import get_project_settings from scrapy.crawler import CrawlerRunner from server.crawler.spiders.recursive_spider import RecursiveSpider @@ -23,43 +25,21 @@ def scrapy_execute(urls, user, title, job_id): reactor.run() -# def execute_scrapy_from_urls(urls, mongo_settings, user=None, title=None): -# id = get_current_job().id -# job_repository.addTask(urls, id, user, title) -# -# pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) -# pool.starmap(execute_scrapy_from_file.execute_scrapy_from_url, [(url, id, mongo_settings, user) for url in urls]) -# pool.close() -# pool.join() -# -# print("Pool Closed") +def scrapy_execute_csv(csv_file, user, title, job_id): + urls = read_urls_from_csv(csv_file) + scrapy_execute(urls, user, title, job_id) + +def read_urls_from_csv(csv_file) -> [str]: + # Read the csv file as pandas df + df = pd.read_csv(csv_file) -# def execute_scrapy_from_flask(filename, file_prefix): -# print('Making new Directory for split files') -# subprocess.run(['pwd']) -# subprocess.run(['mkdir',file_prefix + SPLIT_PREFIX]) -# print("Splitting tmp file " + str(filename)) -# split_cmd = SPLIT_FILE_CMD + '.csv ' + str(filename) + ' ' + str(file_prefix) + SPLIT_PREFIX -# print("Split command " + split_cmd) -# subprocess.run(split_cmd.split()) -# subprocess.run(['ls', '-l']) -# -# print("Starting Pool for file processing") -# pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) -# id = get_current_job().id -# list_files = [(file_prefix + SPLIT_PREFIX + file,id,None) for file in os.listdir(file_prefix + SPLIT_PREFIX)] -# print(list_files) -# -# pool.starmap(execute_scrapy_from_file.execute_scrapy_from_file, list_files) -# pool.close() -# pool.join() -# print("Pool closed. Cleaning up!") -# cleanup_cmd = 'rm ' + filename -# subprocess.run(cleanup_cmd.split()) -# cleanup_cmd = 'rm -r ' + file_prefix + SPLIT_PREFIX -# return subprocess.run(cleanup_cmd.split()) + # Case-insensitive regex match so that + # URLs, urls, Urls, etc. would all work + df.filter(regex="(?i)urls?") + # If there are more than one matches, + # just select the first column + urls = df.iloc[:, 0].tolist() -if __name__ == '__main__': - scrapy_execute(["https://miclin.me"], "miclin@berkeley.edu", "miclin", "id-aijadifj") + return urls diff --git a/server/jobs/interfaces.py b/server/jobs/interfaces.py index 4cc5e5b..268a4b1 100644 --- a/server/jobs/interfaces.py +++ b/server/jobs/interfaces.py @@ -2,7 +2,9 @@ import redis import rq from flask import Blueprint, request, jsonify, g -from server.crawler.run_spider import scrapy_execute +from werkzeug.utils import secure_filename + +from server.crawler.run_spider import scrapy_execute, read_urls_from_csv from server.crawler.tracking import job_repository from .utils import token_required @@ -44,16 +46,26 @@ def all_jobs(): @bp.route("/create", methods=["POST"]) def create_job(): - data = json.loads(request.data.decode("utf-8")) - if "urls" not in data: + if "csv_file" in request.files: + csv_file = request.files["csv_file"] + title = request.form["title"] + urls = read_urls_from_csv(csv_file) + else: + data = json.loads(request.data.decode("utf-8")) + title = data["title"] if "title" in data else None + if data and "urls" in data: + urls = data["urls"] + else: + return jsonify( + message="No URLs or a valid csv file in the request" + ), 400 + + if len(urls) == 0: return jsonify( - message="No URL in the request payload" + message="Empty URLs" ), 400 - urls = data["urls"] - title = data["title"] if "title" in data else None job_id = job_repository.new_job(urls, g.user, title) - queue.enqueue(scrapy_execute, urls, g.user, title, job_id) return jsonify( diff --git a/server/settings.py b/server/settings.py index 97258de..58f1bff 100755 --- a/server/settings.py +++ b/server/settings.py @@ -105,9 +105,9 @@ MONGO_DATABASE = 'crawlerSpider' # database (not collection) name -MONGO_USERNAME = 'admin' # could probably make a "schoolCrawler" user to use here instead +MONGO_USERNAME = os.getenv('MONGO_USERNAME') or 'admin' # could probably make a "schoolCrawler" user to use here instead -MONGO_PASSWORD = 'mdipass' # Replace with actual password +MONGO_PASSWORD = os.getenv('MONGO_PASSWD') or 'mdipass' # Replace with actual password # FILES_EXPIRES = 365 IMAGES_EXPIRES = 365 @@ -115,6 +115,11 @@ IMAGES_MIN_HEIGHT = 150 IMAGES_MIN_WIDTH = 150 +UPLOAD_FOLDER = "uploads/" +UPLOAD_FOLDER = os.path.join( + os.path.dirname(os.path.realpath(__file__)), UPLOAD_FOLDER +) + GOOGLE_OAUTH_CLIENT_ID = os.getenv('GOOGLE_OAUTH_CLIENT_ID') or None FLASK_ENV = os.getenv("FLASK_ENV") or "production" diff --git a/server/templates/home.html b/server/templates/home.html index 3c77320..85d274b 100644 --- a/server/templates/home.html +++ b/server/templates/home.html @@ -2,6 +2,7 @@ + Crawl4All: A Universal Web Crawling App diff --git a/server/uploads/.gitignore b/server/uploads/.gitignore new file mode 100644 index 0000000..86d0cb2 --- /dev/null +++ b/server/uploads/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory +* +# Except this file +!.gitignore \ No newline at end of file diff --git a/setup_guides/mac_local.md b/setup_guides/mac_local.md new file mode 100644 index 0000000..f1346a0 --- /dev/null +++ b/setup_guides/mac_local.md @@ -0,0 +1,95 @@ +## Running the scraping server (macOS) + +### 1. Installation and Setup +First, make sure you have both [Homebrew](https://brew.sh) +```bash +brew -v +``` +and [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/macos.html#install-macos-silent) installed. +```bash +conda -V +``` + +### 2. Redis Installation +To install Redis on Mac +```bash +brew install redis +``` +and start the redis server using _brew service._ +```bash +brew services start redis +``` + +### 3. Conda Environment +Create a Conda environment named `wc-server` using Python 3.10 +```bash +conda create --name wc-server python=3.10 +``` +Activate the environment +```bash +conda activate wc-server +``` +### 4. Install dependencies using `pip` +```bash +pip install -r requirements.txt +``` +### 5. Setup mongoDB and React environment variables +```bash +echo "CLIENT_ORIGIN=http://localhost:3000 MONGO_URI=mongodb://localhost:27000 SERVER_PORT=5000 REACT_APP_SERVER_URL=http://localhost:5000" | xargs conda env config vars set + +conda deactivate +conda activate wc-server +``` + +### 6. Docker +To install the Docker macOS App +```bash +brew install --cask docker +brew install docker-machine +``` +In the project folder, create a folder called `mongodata`, we will use it as the mounting point of our docker container. +```bash +mkdir mongodata +``` +Then we can spawn the docker instance using +```bash +docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v $PWD/mongodata:/data/db mongo +``` + +### 7. Node modules +To install Node.js and NPM +```bash +brew install node +``` +and install all the necessary node modules (such as React) +```bash +npm --prefix ./client install +``` + +### 8. Set up user authorization with Google Sign-In +Replace the Client ID in [`client/src/server-config.js`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/client/src/server-config.js#L5) and [`client/src/settings.py`](https://github.com/URAP-charter/scraping_server/blob/97c303d4f6455a51efe83f16c8d5a8daec272941/crawler/crawler/settings.py#L150) with your own ([how to create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials)). You can enable crawling requests from your IP addresses (if not `localhost`) as "Authorized Javascript Origins" with your Client ID on [the Google Console Credentials page](https://console.developers.google.com/apis/credentials). The current repo uses the Client ID created by [Jaren Haber, PhD](https://www.jarenhaber.com/), which will work for the purposes of testing and developing the crawling server. + +In addition, you must replace the placeholder "clientID" in line 3 of the .env file contained in the main repository with your own Google OAuth Client ID. + +### 9. Run Crawl4All Server +Running Redis Queue, Flask, React Server. In three Terminal windows, navigate to the project folder and run the following commands. + +Terminal 1: +```bash +conda activate wc-server +rq worker crawling-tasks --path ./crawler +``` + +Terminal 2: +```bash +conda activate wc-server +npm run dev::server +``` + +Terminal 3: +```bash +conda activate wc-server +npm run dev::webpack +``` + +At this point, you should be able to see the web interface running on [localhost:3000](http://localhost:3000) diff --git a/setup_guides/ubuntu_vm.md b/setup_guides/ubuntu_vm.md new file mode 100644 index 0000000..d347911 --- /dev/null +++ b/setup_guides/ubuntu_vm.md @@ -0,0 +1,74 @@ +## Linux (Ubuntu) Setup on a Virtual Machine +To set up Web-scraping Framework on a Linux/Ubuntu machine follow the below steps +### Installation and Setup +#### 1. Git Cloning +1. Create and add your SSH key to the GitHub account that you want to use with the VM +2. More information can be found [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) +3. If you have not downloaded Byobu download it using instructions found [here](https://www.byobu.org/downloads) +4. Open your Bash Terminal of choice eg. Git Bash +5. Connect to the VM using the code ”ssh ussername@ip_address" +6. Clone scraping_server Git repo using +7. `git clone scraping_server.git` +8. Make sure that the github repo is the “CSV Upload” Branch with the below code +9. `$ git branch -a` +10. Check that the csv-upload branch is available +11. `$ git checkout ` +12. `$ git branch` +1. Open 3 terminal windows. Byobu use command `Ctrl-a —> c` +#### 2. Redis Setup +1. Setup and start Redis on the VM using the following code: +2. `sudo apt-get install redis-server` +3. ` sudo nano /etc/redis/redis.conf # change 'supervised no' to 'supervised systemd'` +`sudo systemctl restart redis.service` +4. `sudo systemctl status redis` # see if redis is actively running / Ctrl+C to escape Redis +`sudo nano /etc/redis/redis.conf` +`# uncomment the line: # bind 127.0.0.1 ::1` +9. Restart Redis again if you made the previous change +10. Run `sudo systemctl restart redis` +#### 3. Conda Installation: +1. Follow the below instructions from [digitalocean](https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-22-04) +2. Create a Conda environment named `wc-server` using Python 3.10 +3. Run `conda create --name wc-server python=3.10` +4. Activate the environment using `conda activate wc-server` +17. In the scraping_server directory Install dependencies using `pip install -r requirements.txt` + +#### 4. Setup Docket and MongoDB container +1. Make sure that [Mongodb](https://www.mongodb.com/docs/manual/administration/install-on-linux/) and Docker for [Ubuntu](https://docs.docker.com/engine/install/ubuntu/) or [Linux](https://docs.docker.com/desktop/install/linux-install/) are downloaded +2. Run `mkdir mongodata` +`docker pull mongo && docker run -d --name mongodb -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=mdipass -p 27000:27017 --log-opt max-size=500m --restart always -v /vol_b/data/mongodata:/data/db mongo` +3. Check for Environment variables and if they are not already added through Conda, add in the environment variables using the below code +`echo "CLIENT_ORIGIN=http://localhost:3000 MONGO_URI=mongodb://localhost:27000 SERVER_PORT=5000 REACT_APP_SERVER_URL=http://localhost:5000" | xargs conda env config vars set` +`echo "GOOGLE_OAUTH_CLIENT_ID=[980011737294-kriddo55g39bja7timpfk233lm83l8jl.apps.googleusercontent.com]" | xargs conda env config vars set` + +4. `conda deactivate` +5. `conda activate wc-server` +#### 5. Setup Node and NPM +1. Use the following code +2. `sudo npm --prefix ./client install` +3. `npm init` +4. `sudo apt install webpack` + +#### 6. Create three terminal screens: one for Redis, one for Flask, one for React +1. **Redis Window** +2. `rq worker crawling-tasks --path ./crawler` +3. **Flask Window** +4. `npm run dev::server` +5. **React Window** +6. `npm run dev::webpack` +7. Current Project Breaking Error `Error: “webpack 5.69.1 compiled with 7 warnings in 4934 ms”` + + +### Troubleshooting +#### 1. Conda Environment +1. Make sure your conda environment is activated on each of the screens +2. You should see some indication that the conda env is activated like (wc-server) + + +#### 2. Settings: Make sure that your internal settings are correct +1. Redis: make sure settings are correct +2. Mongo: make sure password and username are aligned +3. Docker: make sure you have the right mongo container running + +#### 3. Webpack: Current project breaking error +1. There is an error when running `npm run dev::webpack` +7. The following errors appear `Error: “webpack 5.69.1 compiled with 7 warnings in 4934 ms”` diff --git a/setup_guides/windows_local.md b/setup_guides/windows_local.md new file mode 100644 index 0000000..00bc814 --- /dev/null +++ b/setup_guides/windows_local.md @@ -0,0 +1,84 @@ +## Windows Local Setup for Crawl4All + +### Installation and Settings +1. Install MongoDB +2. Go to MongoDB website and install [MongoDB](https://docs.mongodb.com/manual/installation/) +3. Choose Community Version +4. Create the directory `data/db` on your base computer directory +5. In a Bash Terminal run code to set up [conda](https://docs.anaconda.com/anaconda-repository/admin-guide/install/config/config-mongodb-authentication/) +6. Install Docker +7. Go to Docker website and install the appropriate [Docker](https://docs.docker.com/desktop/windows/install/) for you system +8. Make sure that you have set up the WSL2 backend: more [info](https://docs.docker.com/desktop/windows/wsl/) +9. Open Docker Desktop and ensure Docker engine is running +10. Install Windows Ubuntu +11. Install Redis +12. On Windows Ubuntu Version run `sudo apt-get install redis-server` to download Redis +13. On Windows Ubuntu Version run `sudo service redis-server start` to start Redis server +14. Clone the scraping_server GitHub repo +15. Navigate to comp-[strat/scraping_server](https://github.com/comp-strat/scraping_server) on Github and choose branches +16. Choose CSV-upload branch +17. Clone the CSV-upload branch using `git clone "url for csv-upload cloning"` +18. Change Mongo DB username and password +19. Go to `/web_scraping/scrapy/schools/schools/settings.py` directory +20. Change the following lines for running locally +21. `'schools.pipelines.MongoDBPipeline': 300 # running with Docker and containers MONGO_URI = 'mongodb://mongodb_container:27017' MONGO_DATABASE = 'schoolSpider' # database (not collection) name MONGO_USERNAME = '' # Replace with actual username MONGO_PASSWORD = '' # Replace with actual password` + +### Running the Scraping Server +1. Open the Windows Ubuntu Version and run sudo service redis-server start to start Redis server +2. Open Git Bash or an equivalent terminal +3. Navigate to the `/scraping_server` directory on your machine +4. Create and Activate a virtual environment using `conda create --name wc-server python=3.10` +5. `conda activate wc-server` +6. If Install packages using `pip install -r requirements.txt` +7. Setup environment variables using the following code in Bash +8. `echo "CLIENT_ORIGIN=http://localhost:3000 MONGO_URI=mongodb://localhost:27000 SERVER_PORT=5000 REACT_APP_SERVER_URL=http://localhost:5000" | xargs conda env config vars set` +9. `conda deactivate` +10. `conda activate wc-server` +11. Setup Flask Environment Variables using the following code +12. `echo "FLASK_APP=server FLASK_ENV=development GOOGLE_OAUTH_CLIENT_ID=980011737294-kriddo55g39bja7timpfk233lm83l8jl.apps.googleusercontent.com DEBUG_NO_AUTH_ENABLED=True" | xargs conda env config vars set` +13. `conda deactivate` +14. `conda activate wc-server` +15. Check Docker Desktop and check that “mongodb” container is activated or use the bash commands below to activate +16. `docker pull mongo` +17. `docker run -p 27017:27017 --name mongodb mongo` +18. Open three Bash Windows and run the following Code in each of the respective Windows +#### Window 1: Redis Window +1. In Redis window: +2. Make sure the virtual environment is activated +3. `cd crawler` +4. `rq worker crawling-tasks --path ./crawler` # run Redis +#### Window 2: Flask window +1. In Flask window +2. `npm run dev::server` +#### Window 3: React Window +1. In React Window +2. `npm run dev::webpack # run React server` +3. Navigate the client from your web browser at http://localhost:3000/ + +### Access and Check Data +1. Use the following directions to access, download, and check your data +2. In the Docker Desktop make sure that the “mongodb” container is activated or use the bash commands +3. `docker pull mongo` +4. `docker run -p 27017:27017 --name mongodb mongo` +5. Once the docker-compose command is finished exec into the mongodb_container using (on windows you may have to prefix that command with `winpty`) +6. `docker exec -it mongodb_container bash` +7. In the mongodb_container type `mongo` to activate the mongo shell +8. Use the following code to set up a administrative user +9. `use admin` +10. `db.createUser({user:'siteUserAdmin', pwd: 'passwordprompt', roles:['userAdminAnyDatabase']})` +11. `db.auth('siteUserAdmin', '')` +12. `db` +13. You should see `admin` appear below the `db` command +14. Use command `exit` to escape the mongo shell and `exit` again to escape the docker shell +15. On your local device navigate to `/web_scraping/scrapy/schools/schools/` and run `scrapy crawl schoolspider -a school_list=spiders/test_urls.csv -o schoolspider_output.json` + +### Common Windows Errors +#### Redis Errors +1. Depending on how you are attempting to set up Redis there can be errors that interrupt the running of the scraper, using the method detailed above worked for me but this could cause issues + +#### MongoDb Fail to Connect +1. This is the current issue that seems to stalling the running of the scraper. +2. PyMongo is failing to connect with MongoDb. **Any help on fixing this issue would be greatly appreciated.** + +#### Failure to Download Python Packages +1. At times there was an error which required you to have the correct version of python downloaded. We fixed by upgrading to Python 3.8 diff --git a/wsgi.py b/wsgi.py new file mode 100644 index 0000000..2bf6e0e --- /dev/null +++ b/wsgi.py @@ -0,0 +1,5 @@ +from server import create_app + +if __name__ == "__main__": + app = create_app() + app.run()