How to Use Wget to Download Anything from the Internet

Wget is a command line utility that can be used to download almost anything available on the internet. The catch, is that it should be available over HTTP, HTTPS, or FTP protocols; otherwise Wget won’t be able to download it. There are a number of ways in which Wget can be used, for example, you can use Wget to download every image on a website, or you can download a single file, as well. Wget is even capable of resuming downloads, when possible. So, if you’re wondering how Wget works, and what it can do for you, here is how to use Wget:

Installing Wget

1. Installing Wget on macOS Sierra

Before we get started with using Wget, we will need to install it on our system. Now, fortunately, most Linux distributions come with Wget pre-installed, so you will not have to do any additional work. However, if you’re on a Mac, you will have to install Wget. This can be done easily using Homebrew (download). If you need any help, here are the steps you’ll need to follow:

• First, you will have to install Homebrew on your Mac, in order to be able to download Wget. To do this, just launch the Terminal, and type the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

• After the command completes, Homebrew will be installed on your system. You can now use Homebrew to install packages, such as Wget. In the Terminal, just run the following command:

brew install wget

• Homebrew will automatically start downloading, unpacking, and installing Wget on your system. This step might take some time, depending on your network speed, so be patient. Once the command completes, you should be able to use Wget on your Mac. To check if it has been installed, just use the command:

wget -V

2. Installing Wget on Windows

Installing Wget on Windows takes a little more effort, but it can easily be done. If you want to install Wget on your Windows PC, just follow the steps below:

• First, download Cygwin, and then run the downloaded setup file.

• When it asks for the default download source, choose the option that says “Install from Internet“, and click on “Next”.

• Pick an installation directory for Cygwin, as well as a directory where it will save package files for installation of utilities.

• In the next screen, you’re required to select your internet connection. If you’re using a Proxy, choose the type of Proxy it is, and fill up the details. For most users, though, the choice would be “Direct Connection“.

• You will then be shown a list of mirrors, choose any of these to download Cygwin. If the download fails, you can try again with a different mirror.

• In the next window, search for “wget”, and click on the plus next to “web” in the list that comes up. Here, the checkbox next to “wget” will be selected automatically. Simply click on “Next”.

• The next screen will show you the packages it needs to download. Just click “Next”.

• Once installation is completed, you will be asked if you want to create a desktop icon, and/or add an entry to the start menu. This is completely a matter of personal preference, but I’d suggest you at least add it to the start menu, so you can find it easily.

That’s it. Wget is now installed on your Windows PC. However, you’ll not be able to run it from the command prompt. Instead, you’ll have to launch the Cygwin Terminal, and use Wget from there.

How to Use Wget

Now that we’ve got Wget up and running on our system, let’s explore all the cool ways in which we can use Wget to download files, folders, and even entire websites from the internet. Here are a couple of interesting things you can do with Wget on your system.

Note: I’m using Wget on a MacBook Air running macOS Sierra 10.12.3 beta. However, since the utility we’re using is the same (Wget), the commands will be the same on any operating system you may be using Wget on.

1. Use Wget to Download Files

You can use Wget to download files from the internet with ease. Just use the following command

wget <url>

• Download a File and Save it With a Specific Filename

Files on servers sometimes have the weirdest names, and you may want to download the file, and have Wget automatically rename it to something that makes more sense to you. To do this, just use the following command

wget -o <output_file.extension> <url_to_download>

• Download Only Specific Filetypes

A web server may be hosting a number of different files. So, what if you only want to download all the PDFs on the server, or maybe all the GIFs? Simply use the following command:

wget -r -A pdf <url_to_download>

This particular command was put to use by Mark Zuckerberg in his Harvard dorm room, to download all the images with just one command, and build the infamous website “facesmash.com”.

wget -r -A jpg, jpeg, png, bmp <url_to_download>

2. Download Files from Multiple URLs with Wget

Say you need to download files from a number of different locations on the internet. There may be a PDF file on url1, an important ZIP file on url2, and something else in url3. If you used Wget in the traditional way, you’ll have to run it three times. However, there is a much easier way to do this.

• First, create a file with all the URLs you need to download data from. Make sure you keep each URL in its own line. Save the file, and make sure you remember the name.

• In Terminal, use the following command to download files from all of the URLs:

wget -i <file_name>

where “file_name” is the name of the file you created with all the URLs you needed to download stuff from.

3. Download an Entire Website with Wget

Yes, you read that right. You can use Wget to download the contents of an entire website, as well. Obviously, if the website is too big, this command will take a lot of time to finish, not to mention that it’ll eat up a lot of space on your system. That said, if you want to use Wget to download an entire website, just use the following command:

wget --mirror -p <url> -P ./LOCAL-DIR WEBSITE-URL

This will download the entire website, into a folder “LOCAL-DIR” inside your working directory. The files of the website will be stored in a folder with the same name as the website, thanks to “WEBSITE-URL“.

4. Use Wget as a Web Spider

You can also make Wget act like a web crawler (or a web spider). This is helpful when you want to confirm that the URL for the file you want to download is valid. To use Wget as a web spider, and check the validity of URLs, just use the following command:

wget --spider <url>

This will check if the URL you’ve mentioned exists, and you’ll get a response telling you the results of the check.

• Positive Response:

• Negative Response:

5. Continue Incomplete Downloads

This is another really cool thing that Wget can do. Say you were downloading a file with Wget, but for some reason, you had to shut your laptop down. The next time you want to try downloading that same file, Wget can actually continue the download for you. You just have to ensure you’re in the same folder where you were downloading the file the last time, and then use the following command:

wget -c <url>

When Wget sees that a part of the file is already present in your system, it will automatically continue the download.

Note: If the file on the server has been changed since the last time you tried to download it, then you shouldn’t use this command, because it will result in a corrupted file that you won’t be able to read.

Also, if the server doesn’t allow continued downloads, then Wget will refuse to start the download from scratch. If you want to start the download from scratch, you will have to delete the partially downloaded file from your system, first.

There are some more conditions to using this argument, and you can read more about it on the Wget man page.

6. Limit the Download Speed

If you’re using Wget to download files, and browsing the internet alongside it, you’d probably not want all of your bandwidth to be used up by Wget. Fortunately, Wget includes an argument that you can use to limit the amount of bandwidth that Wget can use for downloading files on your system:

wget --limit-rate=<speed> <url>

Note: Keep in mind that the speed is mentioned in Bytes, and not Bits. So,if you use Wget –limit-rate=20k, it will limit the bandwidth usage to 20 kB, and not 20 kb.

7. Use Wget to Download Files from FTP Servers

If you need to download a file from an FTP server, you can do that with Wget, as well. Simply use the following command to download files from FTP servers:

wget <ftp_url>

• Download Files from FTP servers with Login

IF your FTP server requires a login to allow download files, you can simply pass the credentials with Wget, as well. Just use the following command:

wget --ftp-username=USERNAME --ftp-password=PASSWORD <ftp_url>

SEE ALSO: 30 Mac Terminal Commands to Access Hidden Features

Easily Use Wget to Download Files from the Internet

You can use Wget to download files from anywhere on the internet. However, while Wget almost always works, there are certain servers that prevent access to Wget, in which case, you can try changing the –user-agent for your Wget command. However, that is not a recommended course of action, unless you absolutely know what you’re doing. For the most part, Wget will let you download any file from the internet, as long as you have access to it, and its URL.

So, have you ever used Wget to download files from the internet? Let us know about your experience with using Wget to download files, over the HTTP, HTTPS, and FTP protocols. Also, if you know of some other uses of Wget that you think deserve to be on this list, do let us know about them in the comments section below.