Saves the text of a web page into a new Google Doc, even when the site won't hand its text to anything but a browser.
Uses
Google Docs
Jina AI
Save a hard-to-open web page as a Google Doc
Set pageUrl to blank.Set pageUrl to blank.Set minimumCharacters to 600.Set minimumCharacters to 600.Set maximumCharacters to 20000.Set maximumCharacters to 20000.
Trim pageUrl split by ://[1] and save it as afterScheme.Trim pageUrl split by ://[1] and save it as afterScheme.Split afterScheme by / and save it as hostAndPath.Split afterScheme by / and save it as hostAndPath.Trim the first item of hostAndPath and save it as authorityPart.Trim the first item of hostAndPath and save it as authorityPart.Trim the last item of authorityPart split by @ and save it as pageHost.Trim the last item of authorityPart split by @ and save it as pageHost.
If pageHost is empty or pageHost is nullIf pageHost is empty or pageHost is nullYes
Stop and report a problem: There is no domain name in the address in the 'pageUrl' step, so there is nothing to fetch.Stop and report a problem: There is no domain name in the address in the 'pageUrl' step, so there is nothing to fetch.No
No steps
Join the items from index 1 onward of hostAndPath with / and save it as pagePath.Join the items from index 1 onward of hostAndPath with / and save it as pagePath.
Get the first item of pagePath split by ? and save it as pagePath.Get the first item of pagePath split by ? and save it as pagePath.Get the first item of pagePath split by # and save it as pagePath.Get the first item of pagePath split by # and save it as pagePath.Set pageScheme to http://.Set pageScheme to http://.If pageUrl starts with https://If pageUrl starts with https://Yes
Change pageScheme to https://.Change pageScheme to https://.No
No steps
Set sourceLabel to pageScheme plus pageHost.Set sourceLabel to pageScheme plus pageHost.If pagePath isn't emptyIf pagePath isn't emptyYes
Change sourceLabel to sourceLabel/pagePath.Change sourceLabel to sourceLabel/pagePath.No
No steps
Set directBody to blank.Set directBody to blank.Set directOk to false.Set directOk to false.Set directNote to blank.Set directNote to blank.
If a step below fails, save the error as fetchErrorIf a step below fails, save the error as fetchErrorTry
Fetch pageUrl and save it as response.Fetch pageUrl and save it as response.Trim response.body and save it as directBody.Trim response.body and save it as directBody.Change directOk to response.ok is true.Change directOk to response.ok is true.Change directNote to A plain request to pageHost answered with status response.status.Change directNote to A plain request to pageHost answered with status response.status.On error
Trim fetchError.message and save it as rawReason.Trim fetchError.message and save it as rawReason.Change directNote to A plain request to pageHost failed: rawReason with pageUrl replaced by the address you gave.Change directNote to A plain request to pageHost failed: rawReason with pageUrl replaced by the address you gave.Change directOk to false.Change directOk to false.If directNote contains ://If directNote contains ://Yes
Change directNote to A plain request to pageHost could not be completed.Change directNote to A plain request to pageHost could not be completed.No
No steps
Set minimumLength to minimumCharacters multiplied by 1.Set minimumLength to minimumCharacters multiplied by 1.Set looksBlocked to false.Set looksBlocked to false.If directOk is noIf directOk is noYes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If the length of directBody is less than minimumLengthIf the length of directBody is less than minimumLengthYes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains just a moment (ignoring case)If directBody contains just a moment (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains enable javascript (ignoring case)If directBody contains enable javascript (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains captcha (ignoring case)If directBody contains captcha (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains access denied (ignoring case)If directBody contains access denied (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains attention required (ignoring case)If directBody contains attention required (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains verify you are human (ignoring case)If directBody contains verify you are human (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
If directBody contains are you a robot (ignoring case)If directBody contains are you a robot (ignoring case)Yes
Change looksBlocked to true.Change looksBlocked to true.No
No steps
Set readableText to blank.Set readableText to blank.Set pageTitle to blank.Set pageTitle to blank.Set readVia to blank.Set readVia to blank.If looksBlocked is yesIf looksBlocked is yesYes
If a step below fails, save the error as readerErrorIf a step below fails, save the error as readerErrorTry
Jina AIRead the page at pageUrl with Jina. Save it as reader.Read the page at pageUrl with Jina. Save it as reader.Trim reader.content and save it as readableText.Trim reader.content and save it as readableText.Trim reader.title and save it as pageTitle.Trim reader.title and save it as pageTitle.On error
Stop and report a problem: Neither a plain request nor the reader service could get the text of that page on pageHost. Check the address, and check you are able to open the page yourself.Stop and report a problem: Neither a plain request nor the reader service could get the text of that page on pageHost. Check the address, and check you are able to open the page yourself.Change readVia to Read through the reader service, because the plain request did not come back with the page..Change readVia to Read through the reader service, because the plain request did not come back with the page..No
Change readableText to directBody.Change readableText to directBody.Change readVia to Read with a plain request. The reader service was not needed..Change readVia to Read with a plain request. The reader service was not needed..If readableText is nullIf readableText is nullYes
Change readableText to blank.Change readableText to blank.No
No steps
If readableText is emptyIf readableText is emptyYes
Stop and report a problem: That page on pageHost came back with no readable text at all, so there was nothing to save.Stop and report a problem: That page on pageHost came back with no readable text at all, so there was nothing to save.No
No steps
Set documentTitle to pageHost.Set documentTitle to pageHost.If pageTitle isn't empty and pageTitle is not nullIf pageTitle isn't empty and pageTitle is not nullYes
Change documentTitle to pageTitle.Change documentTitle to pageTitle.No
No steps
Shorten documentTitle to 90 characters and save it as documentTitle.Shorten documentTitle to 90 characters and save it as documentTitle.Set document to nothing.Set document to nothing.Set documentId to blank.Set documentId to blank.If a step below fails, save the error as documentErrorIf a step below fails, save the error as documentErrorTry
Google DocsCreate the document documentTitle. Save it as document.Create the document documentTitle. Save it as document.Trim document.document_id and save it as documentId.Trim document.document_id and save it as documentId.On error
Stop and report a problem: The page was read, but the Google Doc to put it in could not be created. Check the Google Docs connection and run it again.Stop and report a problem: The page was read, but the Google Doc to put it in could not be created. Check the Google Docs connection and run it again.If documentId is empty or documentId is nullIf documentId is empty or documentId is nullYes
Stop and report a problem: Google Docs did not return a document id, so there was nowhere to write the page.Stop and report a problem: Google Docs did not return a document id, so there was nowhere to write the page.No
No steps
Set header to Source: sourceLabelSet header to Source: sourceLabel
readVia
.Change header to header@directNoteChange header to header@directNote
.Set documentText to header plus readableText truncated to a record of kind, op, left, and right characters.Set documentText to header plus readableText truncated to a record of kind, op, left, and right characters.Google DocsAdd text to documentId.Add text to documentId.documentText
Show in the output: Saved the readable text of pageHost to a new Google Doc.Show in the output: Saved the readable text of pageHost to a new Google Doc.