What Web Extractor Does
Web Extractor investigates a supplied web URL and attempts
to locate the information you request using page DOM evidence
and passively observed network JSON evidence.
Start an Investigation
-
Open the Investigate section.
-
Enter the complete target URL, including
https://.
-
Enter the data you want extracted.
You may separate requested fields with commas or new lines.
-
Start the investigation.
-
Use Jobs to monitor the investigation.
-
Use Results to review the extracted
and normalized evidence.
Extraction Request Examples
Comma separated:
page title,Downloads,contact email
Or one requested field per line:
page title
Downloads
contact email
Evidence Types
-
DOM evidence —
visible or structured values found in the loaded page.
-
Network evidence —
JSON responses passively observed while the page loads.
-
Unified evidence —
normalized comparison of DOM and network observations.
Validation Status
-
corroborated —
DOM and network evidence agree.
-
conflicting —
both sources exist but do not agree.
-
dom_only —
only DOM evidence was found.
-
network_only —
only network evidence was found.
-
unresolved —
requested evidence was not found.
Confidence
Confidence and source ranking describe the strength of
observed evidence. They do not automatically choose an
authoritative source.
Web Extractor currently does not automatically resolve
evidence conflicts, select an authoritative source, or
promote a value merely because it has the highest ranking.
Authentication
Web Extractor detects whether a page appears to require
authentication. Existing legitimate browser sessions may
be reused through the site's persistent browser profile.
MFA and CAPTCHA challenges remain user-mediated.
Web Extractor does not bypass them.
Current Authentication States
not_requiredrequired_not_authenticatedauthenticated_session_detectedauthentication_pendingcredentials_rejectedmfa_requiredcaptcha_requiredunknown
Safety Boundaries
- No CAPTCHA bypass.
- No MFA bypass.
- No endpoint replay.
- No automatic conflict resolution.
- No automatic authoritative promotion.
- No production writes.
Browser / API / CLI Interfaces
All three interfaces use the same Dynamic Web Investigator
capability:
-
Browser:
https://webextractor.seebrealty.com/
-
REST API:
/api/v1/investigations
-
CLI:
python3 web_investigate.py
Current CLI --help
The following text was captured directly from the installed
CLI when this GUI Help panel was installed:
usage: web_investigate.py [-h] [--url URL] [--extract TEXT] [--headed]
Dynamic Web Investigator (DWI) / Web Extractor
Site-independent browser investigation runner for opening a target URL, discovering authentication requirements, capturing page evidence, and carrying a requested-data extraction contract.
options:
-h, --help show this help message and exit
--url URL Target HTTP/HTTPS URL. Overrides DWI_TARGET_URL, .env, and
the interactive URL prompt.
--extract TEXT Description of the data to investigate/extract. May contain
multiple lines. Overrides DWI_EXTRACT, .env, and interactive
extraction input.
--headed Run Chromium with a visible UI instead of the default
headless mode. Requires a usable graphical display.
INPUT RESOLUTION
----------------
Target URL priority:
1. --url URL
2. DWI_TARGET_URL environment variable
3. DWI_TARGET_URL from .env
4. Interactive prompt
Extraction request priority:
1. --extract TEXT
2. DWI_EXTRACT environment variable
3. DWI_EXTRACT from .env
4. Interactive multiline prompt
When using interactive multiline extraction, enter one requested item per
line and enter:
END
on a line by itself when finished.
AUTHENTICATION
--------------
DWI automatically investigates whether the target appears to require
authentication.
It does NOT ask:
"Does this site require authentication for the data you need?"
When authentication evidence is detected, DWI may request legitimate user
interaction such as a User ID / Email, Password, or login URL.
Passwords are entered with hidden terminal input and are not intended to be
written into DWI evidence.
DWI does not bypass CAPTCHA, MFA, or access controls.
BROWSER MODES
-------------
Default:
Headless Chromium
Optional:
--headed
The --headed option requires a usable graphical/X display session. On a
headless server it may fail even though normal headless investigations work.
EXAMPLES
--------
1. URL and one extraction request:
python3 web_investigate.py \
--url "https://example.com/" \
--extract "page title"
2. Multiple requested fields from the command line:
python3 web_investigate.py \
--url "https://example.com/" \
--extract $'page title\nvisible page content\nlinks'
3. URL and extraction request from environment variables:
export DWI_TARGET_URL="https://example.com/"
export DWI_EXTRACT=$'page title\nvisible page content'
python3 web_investigate.py
4. CLI URL with extraction request from environment:
export DWI_EXTRACT=$'page title\nvisible page content'
python3 web_investigate.py \
--url "https://example.com/"
5. Environment URL with CLI extraction request:
export DWI_TARGET_URL="https://example.com/"
python3 web_investigate.py \
--extract "page title"
6. Fully interactive input:
unset DWI_TARGET_URL DWI_EXTRACT
python3 web_investigate.py
URL to investigate: https://example.com/
Requested data:
> page title
> visible page content
> links
> END
7. Visible Chromium session:
python3 web_investigate.py \
--headed \
--url "https://example.com/" \
--extract "page title"
8. Authentication-discovery example:
python3 web_investigate.py \
--url "https://www.narrpr.com/" \
--extract $'Grantor\nGrantee\nDocument Number\nRecording Date\nVesting'
ENVIRONMENT VARIABLES
---------------------
DWI_TARGET_URL
Default target URL when --url is not supplied.
DWI_EXTRACT
Default extraction request when --extract is not supplied.
DWI_NONINTERACTIVE
Internal worker-mode flag used by the Browser/API service.
Normally do not set this manually.
OUTPUT / EVIDENCE
-----------------
Each investigation creates an evidence directory containing available
artifacts such as:
summary.json
authentication.json
page.html
page.txt
page.png
The exact directory is printed during the investigation.
AVAILABLE INTERFACES
--------------------
CLI:
python3 web_investigate.py
Browser GUI, when dwi-web.service is running:
http://127.0.0.1:5095/
REST API:
POST /api/v1/investigations
GET /api/v1/investigations
GET /api/v1/investigations/{job_id}
Service health:
GET /health
HELP
----
python3 web_investigate.py --help
or:
python3 web_investigate.py -h
Keyboard
-
Press
? to open Help when focus is not inside
an input field.
-
Press
Escape to close Help.