NASDAQ or S&P500 Components List: How to Download it from Yahoo! Finance
In many are trying to apply Machine Learning Algorithms to the stock market to learn and predict the fluctuation of stocks. Yahoo! Finance is of great help because allows you to download in simple CSV files price and volume of all the stocks going back a few years.
The API is easy to use and to script but unfortunately there is often no list of symbols available to start with. Those lists exist on the sites of the markets (e.g., here is the one for NASDAQ) but they are hard to piece up together.
Luckily, with a could of lines of bash it is possible to create one from the list of components of the NASDAQ or S&P500. The following is the code necessary:
for p in $(seq 0 50 2739); do
wget -O - "http://download.finance.yahoo.com/d/quotes.csv?s=@%5EIXIC&f=sl1d1t1c1ohgv&e=.csv&h=$p";
done >> /tmp/nasdaq.symbols
This script will loop in step of 50 from 0 to 2739 (numbers of element in the components list of NASDAQ) and downloading the list of symbols in each page and concatenating into the file /tmp/nasdaq.symbols.
The URL to download can be found at the bottom in pages list this one and you just need to substitute the &h=50 with &h=$p, which is the iterator variable.
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
Hi Alessio,
That is very interesting. My code knowledge is limited to Excel VBA and to basic R, and the simplicity of the code you posted is very appealing.
Can I ask a non-technologist’s question? What language is it written in, and what platform(s) will it run in? Thanks,
Guy
Hi Guy, this code is written in “bash”, which stands for “Bourne Again Shell”. Proffered by GNU, it is a basic scripting language on *nix-styled platforms (Linux, Unix, Solaris, OS X…), conceptually somewhat like Windows batch files.
Alessio’s program executes another program called wget, which is useful for downloading singe files, or entire subsections of a web or ftp site. For more info, see references below.
Hope this helps,
Richard
bash – http://www.gnu.org/software/bash/
wget – http://www.gnu.org/software/wget/