This example shows an FTP server but the same works for HTTP server

/usr/bin/wget \ –verbose \ –mirror \ –wait=2 \ –random-wait \ –no-host-directories \ –cut-dirs=3 \ –directory-prefix=/home/ftp/doom/idgames_mirror \ –dot-style=binary \

So ^that is how you can efficiently download all the things from a HTTP/FTP mirror (only if you have permission now!!). This is what I used to copy the idgames archive to my mirror at Michigan Tech, PSGNet.

I would look up what each of these options means on the wget man page if you aren’t sure.
More here.

Mario Loria is a builder of diverse infrastructure with modern workloads on both bare-metal and cloud platforms. He's traversed roles in system administration, network engineering, and DevOps. You can learn more about him here.