Podcasts Technology Syntax - Tasty Web Development Treats The Undocumented Web: scraping, private APIs, proxies and alternative solutions

The Undocumented Web: scraping, private APIs, proxies and alternative solutions

Published: July 25, 2018, 1 p.m.

b'

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo\\u2019s Freelancing & Beyond \\u2014 Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at\\xa0https://studywebdevelopment.com/freelancing\\xa0and use the coupon code \\u201csyntax\\u201d at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at\\xa0freshbooks.com/syntax\\xa0and put\\xa0SYNTAX\\xa0in the \\u201cHow did you hear about us?\\u201d section.

Show Notes

3:05 - What Is the undocumented web?

  • When traditional methods don\\u2019t give you what you want
  • Wild West

6:10 - Undocumented APIS

17:34 - Faking

  • Faking User Agents
    • Sending headers from requests
    • Free Blackberry wifi on plane

23:52 - Scraping

  • What is craping?
  • Cheerio\\xa0- jQuery like node package

27:30 - Automation

33:30 - Proxies

  • Need a fresh IP address
  • Need a different country
  • Side note: Canadians buying stuff in USA M4S1P1 === 41100
  • API throttling

39:00 - Tips and Tricks

  • Multiple API keys
  • diskDB\\xa0- Text files as databases
  • Search Github for API keys
  • Inspect Element and Delete Overlays
  • Delete Cookiesokay

44:45 - Stories

\\xd7\\xd7\\xd7 SIIIIICK \\xd7\\xd7\\xd7 PIIIICKS \\xd7\\xd7\\xd7

Shameless Plugs

Tweet us your tasty treats!

'