Linux multiple threads download from file with wget
Assume we have a file that have all the urls to download.
sed 's|^|https://data.commoncrawl.org/|' wet.paths | xargs -n 1 -P 15 wget --timeout=1800 --retry-connrefused --waitretry=10 -nc &
Previous post: How to config logging output to file and stdio/console with basicConfig() in one line?
Next post: KeyCloak vs Authelia