Do you remember the wild west phase of the Internet, when every website designer just did their own things, and pages would be filled with mismatched colors, weird UI choices, and stretched-out images? What a time to be alive. Moreover, think back to how those websites looked if you accessed them from a phone or tablet. Navigation wasn’t just a chore, it was downright painful. Everything is much more streamlined now, anchored in good UI practices, and optimized for all kinds of screen sizes. We have Javascript to thank for that last part. It’s the magic language that turns boring static pages ..read more

It’s easy to throw all APIs together into one big pile and say they’re all the same thing. But that’s not really accurate. Sure, they’re all Application Programming Interfaces, and all of them bridge the gap between different products or services without the need to rewrite them for proper integration. While everyone is free to design and build their software however they see fit, almost all APIs conform (at least partially) to clearly defined styles. These guidelines are meant to keep uniformity and maintain the API’s primary goal: to simplify integration and communication. If you’re interes ..read more

In web scraping, you’re either using proxies or doing a poor job extracting data. Sure, you may not need a backconnect proxy with thousands of datacenter and residential IPs, but you’ll at least want some basic proxies. The big problem for hobby or beginner scrapers is that IPs usually cost money. Luckily, there is such a thing as free proxies. In fact, there are quite a few options to choose from. We’ve explored some of them and chose the best 7 so that you only get the quality, at least as far as free proxies go. Actually, before we get into the list, we should go over some details. Free pr ..read more

We live in a digital age, and we have access to a ton of data. Using this information, we can better understand how we work, play, learn and live. Wouldn't it be nice to be able to get more targeted information on a particular topic? With this article, we will demonstrate how to use BeautifulSoup and WebScrapingAPI together to build our own crawlers for gathering targeted data from websites. A word of warning: scraping copyrighted content or personal information is illegal in most cases. To stay safe, it’s better to get explicit consent before you scrape any site, especially social media site ..read more

Have you ever wondered how businesses keep track of the competition? Of course, double agents, private investigators, and binoculars may sound fun, but real life isn’t like the movies. Actual competitor analysis is a tedious process that’s better left to the software. This way, you can focus on delivering the most value to your customers. Nowadays, your competitors already have all the information you need in the open. With social media gaining a lot of traction in the last couple of decades, businesses are trying to be more present in their customer’s life by creating an online presence. The ..read more

Let me paint a picture for you. You just launched your rare retro video game cartridge online shop, and you don’t really know how to handle the product pricing. Setting the price for a product is always a challenge: if the price is too low, you cut from the profit margin. If the price is too high, customers will not be interested in the product and will buy it from a competitor. Decisions, Decisions. What can we do? Given this problem, you have two solutions: Spend hours every day to research the prices from competitors Be smart about it and create an automatic script that does this mundane ..read more

Rather than using commercial tools, many developers prefer to create their own web scrapers. While available products have more fleshed out features, we can’t deny the results these bots can bring or the fun of making your own. In the following article, you’ll find out the steps you have to take to build your own web scraper using Node.js and Puppeteer. We’ll code an app that loads a website, snaps a screenshot, log in to the website using a headless browser and scrape some data across multiple pages. Your app will grow in complexity as you progress. An overview of Puppeteer Google designed P ..read more

APIs come in many shapes and sizes. While plenty of developers have APIs to thank for making their jobs more manageable in a hundred different ways, not many actually take the time to learn more about these interfaces. We get it. Time is limited, and understanding what defines a REST API might not be useful for everyone. We won’t deny that, but there’s a reason why just about anyone should learn this: it’s damn interesting. Plus, if you want to design or work with an API, it pays to know more about it. So, in this article, we’ll take a look under the hood of REST APIs and understand how their ..read more

Let's not sugarcoat this, as the years drag on, the Internet's data volume will only continue to rise. The situation is beyond anyone's control, but is that really a bad thing? Particularly in the past decade, web scraping has gained enormous popularity. To succeed in today's business world, companies need proper strategies, which require ample information within a short amount of time. Plus, it could be a neat way for developers to improve their coding skills AND help the company. If you play for team Ruby but don't have much to do with web scraping, this article will provide you with a new ..read more

Web scraping entails paying close attention to a vast number of obstacles. Proxies should be the first solution you think about when collecting data off the Internet. At least, that’s the first step if you don’t want to run into problems from square one. We have already expanded on the many kinds of proxies available: residential, datacenter, mobile, or rotating. In this article, however, we want to show you how to get both quality and quantity with proper research on dedicated and shared proxies. Finding the best solution for your project can be tricky. After all, there are plenty of options ..read more