Angular get html from url

I'm trying to get some url html content from angular But all I get is Cross-Origin Request Blocked.

Here's the code I was trying to use.

this.http.get('http://www.google.com')
   .subscribe(
     data => {
       console.log(data);
     }
   );

I expect to find in the console

   ...

asked Apr 24, 2019 at 14:25

0

try this:

 import { HttpHeaders } from '@angular/common/http';

 const httpOptions = {
 headers: new HttpHeaders({ =
 "Access-Control-Allow-Origin" : "*" 
 })}; `

this.http.get('http://www.google.com', httpOptions )

or install extension enable cors for your browser

answered Apr 24, 2019 at 15:53

1

I thought I would try to share a small solution for a funny problem with you (notice how I share it personally with ‘you’ — If you are here I welcome you and am pleased to be able to help — If you are not here no one will ever notice that I used the term ‘you’. Clever huh ;)

The other day we had to implement new terms in our on-boarding form. The catch was that the terms were hosted on an external website. Our site is built in Angular 11 and this seems like a trivial issue that could be done in a matter of seconds. Which it can if you omit the UX part.

The first suggestion was merely an a-element pointing to the URL with the terms:

  {{ ‘I have read and accepted’ | translate }}


  {{ ‘terms’ | translate }}

The main problem with this is that your users are now thrown to an external website in the middle of the on-boarding process. Furthermore there is no easy way back; you cannot check ‘Accept’ on the external page, you have to close the page again manually and get back to the form and so forth. Just not good UX.

The solution — Web scraping

Why not scrape the terms text from the external website and insert it in a modal instead? Let’s do that.

First we need to know where and what we are trying to get. The ‘where’ is just the URL for the website.

We will use async/await (that’s ES7 I believe) and first get the entire web page HTML:

(async () => {
  const response = await fetch('https://your/external/terms/');
  const text = await response.text();
  console.log(text);
})();

The console will then print out all the HTML — But with a catch: This is a string!
Tip: You can do this directly in the browser by going to the external URL, open the console in developer tools, paste in the above code and hit ‘Enter’ :)

For the ‘what’ go to said website, right click the text and choose ‘Inspect element’. This is what it looked like in my case:

The DIV marked in blue is what I need. The closest I get to something I can directly address is the sibling DIV with the ID=”sidebar” so let’s try that.

Our challenge here is that we cannot navigate the HTML in string format — We need a DOM so let’s format the string ‘back’ into HTML.

I chose a well used and maintained npm package node-html-parser and added import { parse } from ‘node-html-parser'; to my component. So let’s put that to work.

(async () => {
  const response = await fetch('https://your/external/terms/');
  const text = await response.text();
  const terms = parse(text);
})();

terms is now the string converted into HTML which we can now traverse so we will take out the DIV #sidebar, go to the next sibling and grab the inner HTML.

htmlTerms: string;(async () => {
  const response = await fetch('https://your/external/terms/');
  const text = await response.text();
  const terms = parse(text);
  this.htmlTerms = terms.querySelector('#sidebar').nextElementSibling.innerHTML;
})();

First we took out all the HTML as a string with response.text() but in order to get to the juicy parts we parsed it back to HTML with parse(text) and traversed this newly created DOM with terms.querySelector('#sidebar').nextElementSibling.innerHTML;.
You now have a string again containing the exact content you need and can place that directly in your own component view with the property htmlTerms .

Note that Angular has built in XSS and that you might need more manipulation than shown above. Also you will loose the HTML if the external webpage is changed related to your scraping code.
So maybe you need some error handling just in case ;)

I chose an interactive modal for the terms (in German ;) but you can implement it anyway you want.

You can of course manipulate the code or add your own styling (or their styling) but the more you manipulate the more can break if the original website is changed so I prefer KISS.

Have fun.