Configure wget with proxy

Posted on

This is an article consists information about to use wget utility behind a proxy server. Eventually, if in one case there is no proxy server which is needed to be passed in order to directly connected to the internet, wget can work normally as is without any further configuration.

But the story will be different if there is a proxy server which is needed to be bypassed in order to directly connected to internet. As it is realized to be one of popular Linux operating system distribution tool to download file and based on the manual page it is said that it is the non-interactive network downloader, a proxy server will be a hindrance if wget cannot be configured properly to adapt with the proxy server.

This is what happened if trying to download file using wget behind the proxy server for an example downloading file named CentOS-7-x86_64-GenericCloud-1511.qcow2 from the available URL of http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2 as shown below :

root@hostname:~# wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2

This is the output displayed when it is executed in a command-line within a bash prompt :

root@hostname:~# wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2
--2016-09-06 10:11:03--  http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2
Resolving cloud.centos.org (cloud.centos.org)... 162.252.80.138, 2604:4500::2a8a
Connecting to cloud.centos.org (cloud.centos.org)|162.252.80.138|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://proxy.office.xxx.xxx/?cfru=aHR0cDovL2Nsb3VkLmNlbnRvcy5vcmcvY2VudG9zLzcvaW1hZ2VzL0NlbnRPUy03LXg4Nl82NC1HZW5lcmljQ2xvdWQtMTUxMS5xY293Mg== [following]
--2016-09-06 10:11:04--  http://proxy.office.xxx.xxx/?cfru=aHR0cDovL2Nsb3VkLmNlbnRvcy5vcmcvY2VudG9zLzcvaW1hZ2VzL0NlbnRPUy03LXg4Nl82NC1HZW5lcmljQ2xvdWQtMTUxMS5xY293Mg==
Resolving proxy.office.xx.xxx (proxy.office.xxx.xxx)... xxx.xxx.xxx.xxx
Connecting to proxy.office.xxx.xxx (proxy.office.xx.xxx)|xxx.xxx.xxx.xxx|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

As it can be seen from the above output, the download process failed with the following message of ‘401 Unauthorized Username/Password Authentication Failed’  :

Resolving proxy.office.xx.xx (proxy.office.xxx.xxx)... xxx.xxx.xxx.xxx
Connecting to proxy.office.xxx.xxx (proxy.office.xxx.xxx)|xxx.xxx.xxx.xxx|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

To be able to use wget to connect with proxy in Linux operating system distribution the following file needs to be edited which is located in /etc/wgetrc.

root@hostname:~# vim /etc/wgetrc

Below is the entry which is needed to be added :

http_proxy=http://url.proxy-address.xx.xx:port-number
proxy_user=user_account_to_connect_proxy
proxy_password=password_account_to_connect_proxy

http_proxy : it is the variable which is used to define the HTTP proxy server address
http://url.proxy-address.xx.xx : it is the URL address of the HTTP proxy server
port-number : it is the port number which is used by the proxy server to process request, normally for an example 8080
proxy_user : it is the variable which is used to define user which is used to connect to the proxy server
user_account_to_connect_proxy : username which is used to connect to proxy server as value to the variable proxy_user
proxy_password : it is the variable which is used to define password which is used to connect to the proxy server
password_account_to_connect_proxy : password which is used to connect to proxy server in accordance with the username attribute. 

After editing file stored in /etc/wgetrc and if the assumption where the additional entry specified is true and valid, this is when wget used with a proper configuration to handle to bypass proxy server :

root@hostname:~# wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2
--2016-09-06 10:11:18--  http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1511.qcow2
Resolving internet.proxy.office.xxx.xxx (proxy.office.xxx.xxx)... xxx.xxx.xxx.xxx
Connecting to proxy.office.xxx.xxx (proxy.office.xxx.xxx)|10.248.0.20|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 899874816 (858M)
Saving to: ‘CentOS-7-x86_64-GenericCloud-1511.qcow2’

CentOS-7-x86_64-GenericCloud-1511.qcow2   100%[==================================================================================>] 858,19M  86,0KB/s    in 85m 32s     ] 509,85M   167KB/s    eta 34m 56s

2016-09-06 11:36:51 (171 KB/s) - ‘CentOS-7-x86_64-GenericCloud-1511.qcow2’ saved [899874816/899874816]

You have new mail in /var/mail/root

The above output shown the download process is success after editing /etc/wgetrc file with additional valid entry.

One thought on “Configure wget with proxy

Leave a Reply