Python Forum » Web Scraping & Web Development
10 FOLLOWERS
The Python Forum is a place to discuss Web scraping-related questions and other topics related to web development. Share your web dev-related questions about Web frameworks such as Django, Flask, WSGI, etc., and get answers from Python experts and other web developers.
Python Forum » Web Scraping & Web Development
1w ago
I am working on a script about role-based authorization by using Flask. I am trying to run by gunicorn on Linux Ubuntu 18.02. However, every time I call the script, it does not proceed. Script, my prompt and the output are below.
from flask import Flask, redirect, url_for
from flask_login import LoginManager, UserMixin, login_user, logout_user
from functools import wraps
from werkzeug.middleware.proxy_fix import ProxyFix
app = Flask(__name__)
# Simulated user database
users_db = {
'alice': {'password': 'password123', 'role': 'admin'},
'bob': {'password': 'password123', 'role': 'use ..read more
Python Forum » Web Scraping & Web Development
2w ago
Hello, I'm trying to get a a script to work but I'm having some issues. I need a bot to scrape the "Latest" post which has been posted on specific forum thread which then sends it to a discord channel.
I keep getting 401 error. I then changed it and added login credentials however even after that it failed to scrape the posts and send them to a discord. Any help would be much appreciated.
import discord
from discord.ext import commands
import requests
from bs4 import BeautifulSoup
from urllib.parse import urlencode
# Discord bot token
TOKEN = 'Yes I have it'
# Forum URL to scrape
FORUM_UR ..read more
Python Forum » Web Scraping & Web Development
3w ago
Hi, let me start by saying I'm a beginner.
I wanted to try my hand at this adventure, convinced that with ChatGPT 4 I would be able to do it.
But here I am.
I need to create a management software that can be managed from a browser.
Management software to be implemented on a Raspberry PI 5 with OS having Raspberry PI OS installed as the operating system.
Purpose of the management system:
Compare 2 images and if they are similar for a percentage equal to or greater than that required by the project, the project is approved, otherwise you are asked to insert a new image.
Management software st ..read more
Python Forum » Web Scraping & Web Development
1M ago
I have developed a website using python Flask which is hosted on IIS and accessible to my team. My question is for security reasons how to package the python files, templates and static folder into a single executable file or .dll file and the same will be hosted on IIS server. Is it possible to directly host an executable (.exe) file on the IIS server? Please suggest an idea or reference document.
Note: I have tried using Pyinstaller to create a single executable, but it doesn't seem to run on IIS ..read more
Python Forum » Web Scraping & Web Development
1M ago
Hi all
I want to upload my Django project folders in Github. But I am unable to upload folders , I tried Git desktop also I did not get solution and I tried git blash also the code is not working like init , gith add . These codes are not working in my laptop
Please anyone can help me
Give me any advice to deploy my project in another free website to get free hosting . I tried Netlify also It is not supporting for Django without Github ..read more
Python Forum » Web Scraping & Web Development
1M ago
My PC is blocked from a video streamer site,
previously I could do this:
using selenium to extract the m3u8 file, Then using FFmpeg to download and convert the m3u8 file to mp4
now, if I access m3u8 using FFmpeg, the response is :
"HTTP error 403 Forbidden, Error opening input: Server returned 403 Forbidden (access denied)"
I can still access the site but only the first page, I can still play the video on that page,
But if I open a page on the same site, the result :
"Sorry, you have been blocked, You are unable to access thesitename.com"
now my question :
is possible to download m3u8 using ..read more
Python Forum » Web Scraping & Web Development
1M ago
I am attempting to put together a lightweight script that will simply scrape a url for 1x1 images or transparent images and store the results and img file for review on a data project. I am already using BS4, PIL, and BytesIO. If it runs w/o error, then I cannot locate the stored images in the working directory, but mostly it errors. Thank you ..read more
Python Forum » Web Scraping & Web Development
1M ago
I'm trying to read the below xml generated in a browser using Beautifulsoup:
<?xml version='1.0' encoding='UTF-8'?><IM415 xmlns="http://www.ros.ie/schemas/customs/IM415">
<Declaration>
<MsgType>H1</MsgType>
<DeclarationType_1_1>IM</DeclarationType_1_1>
<AdditionalDeclarationType_1_2>A</AdditionalDeclarationType_1_2>
<LRN_2_5>NIK243186_16KAkj</LRN_2_5>
<ValuationInformation>
<InvoiceCurrency_4_10>EUR</InvoiceCurrency_4_10>
<InvoiceAmount_4_11>5000</InvoiceAmount_4_11>
<InternalCurrency_4_12>EU ..read more
Python Forum » Web Scraping & Web Development
1M ago
Hello there,
First of all, I am not an expert for Python, but I rather learned it AD HOC. Namely, I have a script that scrapes name, sizes and prices for each size for products. I create excel file that the script reads, open links, gets the data, and saves it in a new excel file. Now, the issue is that the script is not able to click on a popup window (cookie) that shows up when run on server and accept the cookies - when I run it on my PC, the script works fine.
Here is the code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver ..read more
Python Forum » Web Scraping & Web Development
1M ago
Hi all
I have 3 html pages
index.html - It contains form
result.html - when you click on submit in index.html it will display result.html page in this page we have context i.e {{result}}
I want to show this context in database
when we click on submit in index.html the data will store in database for this i created si class in models.py
Please anyone can help me ..read more