This example shows an FTP server but the same works for HTTP server
/usr/bin/wget \ –verbose \ –mirror \ –wait=2 \ –random-wait \ –no-host-directories \ –cut-dirs=3 \ –directory-prefix=/home/ftp/doom/idgames_mirror \ –dot-style=binary \ ftp://ftp.fu-berlin.de/pc/games/idgames/
So ^that is how you can efficiently download all the things from a HTTP/FTP mirror (only if you have permission now!!). This is what I used to copy the idgames archive to my mirror at Michigan Tech, PSGNet.
I would look up what each of these options means on the wget man page if you aren’t sure.
More here.