10.9: New York Times API and JavaScript – p5.js Tutorial

Posted by


Hello. In this video, I’m going to
look at The New York Times API, which is an API that you
can query to get information about articles in The New
York Times, shockingly. Quite a surprise. Yeah. So again, I’m not
suggesting that this is the be all, end
all of APIs to use, but this is a nice case study
of finding an API that’s out there, how do you
navigate its documentation, how do you figure out
how to form an API query and look at that data
and make something happen on the screen in a web page
using JavaScript, HTML, and CSS. And this thing over
here above my head– I just would rather
have that as a hat– called p5 GS. So the first thing, once
again, as in the previous video where I looked at Wordnik. The first thing is I
can spend a lot of time without even writing any
code or looking at any code because the first
thing I need to do is just figure out, like, huh? There’s an API? Like what? So one thing, you’ll
see, I’m already here. But I’m at
developer.nytimes.com. So once again, you’re
at the mercy of the API. How good is the documentation? Are there examples? Is there an online tool that
helps you navigate the API? And again, this is a case
where there is a good one. And this is that question, like
you went to The New York Times website, and you found all this
data, and you want to use it. Is there an easy path
of that, or do you have to take some
sort of side road? In this particular video, I’m
showing you the easy path. And in another video, we’ll
look at some side roads to how you might scrape or
parse data that isn’t presented in a API, json friendly format. So all of that aside, here I am. Welcome. Get started. This is nice, friendly language. It makes me happy that someone
is welcoming me to the page, and I could get started. So the first thing you’re
going to need to do, by the way if you’re doing this on your
own, is request an API key. So again, you’ve got to sign up
for an account, get an API key. I don’t believe– actually
I don’t know this one. I have no idea what
I’m talking about all of a sudden, which
is most of the time. But I don’t know whether
there’s any threshold of querying this API
where you suddenly have to pay for its use. But certainly, what I’m
going to show you today, getting a little bit
of data here and there, is something you
can do for free. This is a great thing. This is something that if
you see is good news for you. You’re looking for
an API tool, meaning you can figure out
how the API works through an interface
in the browser. It’s all the same exact
thing with Wordnik. So I could click around
and look at various things, available APIs, but I’m
going to go here directly to that API tool. And here you can
see API console. Actually, before I go that,
let me click to available APIs. So you can see here,
there’s a lot of APIs. There’s event listings and
geographic data, most popular API, movie reviews. There’s lots of different
sub-APIs, different paths within the larger API. I think I’m going to stick
with, for simplicity, just this article search API. So interesting, this
is a nice data source because it actually
goes back to 1851. So I can query The New
York Times to ask for or I can search. I can say, give me
all the articles that have the word “rainbow”
in them or the word sea-bow. There’s a such
thing as a see-bow as I learned in a previous
video or porcupine, whatever. Pick your word of choice. You can find the articles. So I’m want to
start with that one. And I could click on it. And here’s some documentation
of how it all works. But once again, I’m going
to prefer to go back and go to this API tool. And here I am. Hi. Moment of technical
difficulty, but I’m back, and I am on the API console. So the first thing that
I’m going to do here is look at how the
API console works to figure out what
the query is I want to make to The New York Times. So I’m in the
article search API, but, of course, I could
change to a different one. And what I want to
do is click here. This is what’s
known as endpoint, being the point
of end of the API where I’m going
to get something. But really, I think endpoint
is referring to search or maybe that’s the method. It doesn’t really
matter this terminology. The point is I want
to search the API, and this is the
path for doing that. So I can click here, and
I can see now, again, I have an interface. And yes, I want the format
to come back as json. That’s going to
work well for me. And then q is the
search query term. So what’s the
query term that I’m looking for articles that
contain that particular word? So I don’t know. I really should think
of something different, but I’ll just keep
using rainbow. By the way– no, never mind. And then, there’s other things. I could filter the search. Oh, there’s a begin
date and an end date. So I could restrict only
getting articles from like 1957 with the word rainbow in it. I could sort what I get
back by a different order. So all that stuff
is interesting, but it’s also all
optional right now. I’m going to leave it out. So I encourage you
to look through this. Try it on your own. Add different parameters
to the API search. But the point is I
want to hit this now. And once again, the
most beautiful button in the internet is the Try
It button because that’s going to let me just try the
API without having to write the code to see how it works. Now one thing here you can
see, The New York Times will often give you some results
with a key known as sample-key. I’ve overused the sample key. So I tried this a
minute ago, and it’s going to tell me, no, sorry. You’ve overused the sample-key. But I have registered and
logged in for my own API key. So I’m not going to include
this process in the video. If you have trouble, ask a
question in the comments. Maybe I can help you. But once you’ve made
an account, logged in, you will get your own key. And I’m going to take that key,
and I’m going to go back here, and I’m going to
paste it in here. And then I’m going
to hit Try It. And once again, we can see
this is exactly what I want. The API tool formed
the API query for me. And also, we can scroll down. You can see it’s
showing me the data. But this is a bit awkward
to look at it here, so what’s a little bit easier
for me to do is grab this URL. Once again, open up a
new tab in the browser, paste it into the
browser, and now I can start to look at the data as
it looks in the browser itself. So I can start now. So first of all, it might
look like this to you. You might get this result.
I have a particular browser extension which formats
json for me nicely. I demonstrated in
an earlier video. I think if you just Google
Chrome extension json formatter, you’ll find one. But even so, this is a good
example of how a lot of APIs will give you back
a ton of data. So really there’s some
detective work involved here in figuring out how
to find the data. So let’s say what we want
to do is show on screen the headlines for
all of the articles that contain the word
rainbow and maybe even a short snippet paragraph from
those particular articles. So let’s think about
how we might do this. First of all, one thing
I often like to do is start minimizing these. You can see that really
the API is just giving me back an object with a
response property, a status property, and a
copyright property, which is important to note that
we’re getting some copyrighted material here. So I can unpack this
response property. There’s a meta property
and a docs property. Meta probably has
some information about the API query, how
many calls I have left, that sort of thing. Docs, which has 10 items– I know this in advance– that’s giving me 10 articles. So most APIs won’t give you
back all 10,000 articles all at once. They’re going to give
you 10 at a time. And there’s a way of
paging through them, but that’s a bit beyond the
scope of this particular video. But what I want to do
is here look at docs, and docs is an array. So you can see these are
each element of the array. So each one of these
elements of the array is one particular article. So in this particular article– now I get to zoom back
out here and look– multimedia, headline–
there we go– keywords. There’s so much information–
publication date, byline. You can see some
things are right there, like snippet,
lead paragraph, abstract. I’m kind of interested
in the headline, which is now in another object. So I can unpack that and see
main, kicker, print headline. So we can see, I can spend all
day here looking into the API. But I can kind of
imagine now if I want the headline
is response.docs index, 0 for the first
one, dot headline.main. So I weirdly feel like I
have that memorized now, probably because I’ve
used The New York Times API a bunch of times. But this is the
thing that you’re going to be needing
to take notes, practice, going to make
a mistake, try it again. This is the thing you’re going
to want to do on your own. But now, we are ready to go
back and start to write code. We’ve looked at the API website. We’ve looked at
the documentation. We’ve used the tool. And we’ve got a working
path to getting data. And so now I’m going
to go back to p5. And I’m going to say just
at the beginning loadJSON, and I want the API URL– I’ll have that as a variable–
and a callback called gotData. So I’m going to make a
variable, which is the URL, and I’m going to paste
this URL right in here. Now for expediency
here, I’m not going to go through the steps of
breaking that URL into parts. I’ve done that several times
in several other videos now. And, in fact, I would say
probably the exercise for you at the end of this video is to
have the user input the search word and then get
particular articles back with that search word. Right now, I’m just going
to get rainbow articles. But the first thing, what
I need to do right now is write this gotData callback. And remember, the gotData
callback needs an argument. That’s where the
data from the API will fill that
variable with the data. And I can just look at
it to make sure it’s working in the console. I’m not really going to
look at in the console, but I want to just hope
that I see something there. And you can see, there
is something there. So hopefully, we can
feel pretty confident that what’s there is
exactly the same as here. So now let’s think
about this again. Response.docs index
0 dot headline.main. Let’s work with that. Data.response.docs index
0 dot headline.main. Did I dare possibly
get that right? Let’s try it again. Let’s run it. Looks like I got that right. Let’s look at the second article
to see what the headline is. I’m changing that number 0– “Hollywood Dreams”. That makes sense that
you might have a rainbow in your Hollywood dreams. My Hollywood dreams
consist of YouTube videos about data and API, which is
sort of a strange Hollywood dream some might say. I don’t know where
I’m going with this. Let’s stop now. Where are we? Oh, I’ve only been
doing this– no, no. I forgot I have
two sections here. Anyway, I’m back. So blah, blah, blah. What do I want to do with this? Well I want to make something
appear in the browser. Now there’s a lot of
interesting applications here. And maybe I’ll get
to some ideas for you towards the end of this video. But right now,
let’s at least make it loop through and show me
every headline and perhaps also a snippet. So let’s go back to this. You can see snippet is actually
just response.docs index 0 dot snippet. Snippet being like a short
paragraph or a snippet from the article to give
us some more information. So one thing that happens
here is that I have an array– data.response.docs. And you know what I
think can be useful? This is a useful technique. If I just make up a variable
called articles and say articles is data.response.docs, I’m going to need it every
time I try to look for data. I’m going to say
data.response.docs blah, blah, blah, data.response.docs
da, da da, da. If you remix this, it has kind
of like a rhythmic quality to it– data.response.docs
da, da da, da. But I’m going to need to do
that over and over again, so it’s kind of useful if I just
store that in a variable called articles because now I can say
things like what I want to do is have a for loop. I goes from 0 to
articles.length I plus plus. And I could say something like– whoops, let me leave that
down here for reference. I don’t need to say
data.response.docs headline main. I just thing to say
articles headline main. Well, what I need to say is– and I’ll say createP articles
index i.headline.main. So this should now loop
through every single object in that docs array– each one having a headline
object and each headline having a main being
the main headline. So if I run this now,
you’re going to see, here we go, I’ve got Rainbow
international, Painbow Charms– Painbow? What’s a Painbow? That sounds terrible. No, no, no. No painbows– rainbows. East Village Birdman
abandoning his nest of CDs. Anyway, there’s some interesting
articles about rainbows. But maybe what I want to do
actually is say createElement. I’m going to make an H1
element for every headline. And then I’m going
to just quickly– I’m kind of like trying
to wrap this up here. I’m going to make
a paragraph element for every snippet,
which would be articles index i dot snippet. So again, there’s a
lot of data there– articles index i
dot headline.main, articles index i dot snippet
make an H1 make a paragraph. And now when I look
at this, we can see I’ve now got like my own
version of The New York Times right here of a bunch of
articles and little snippets. And right now, if I wanted to
change what it’s searching for, I’ve got to go into
my code into this URL. I could change it to porcupine,
and I could run it again. And you can see, I’ve got
now some prickly in name but refined on the table. I’ve got some articles
about porcupines. So here’s what I
would say to you. First of all, just
showing you the beginnings of what you could do here. So one thing that I think
would be a good exercise is to make this work but not
have to go type into the code to change the word. Add a text input field. Add a button. When the user enters something
in that text input field and clicks the button,
you get the articles with that search term. Another thing you
might think about doing is, can you add query
string arguments? The only query argument
I have right now is q equals porcupine. But if you go back to the
tool, there was a begin and an end in terms of dates. So could I get all the articles
from a particular year? Could I even have that year
be selected by the user? That’s something you might try. Another thing you
might look into, and I might need to add some
link or something in this, is that The New York
Times will give you the frequency that a word
appeared in The New York Times for a given year. So you can look at
interesting trends like when did the
word computer start getting used often in articles. This is a very
totally trite example, but you could think of something
like how many times the word war used versus the word peace. So anyway, you’ll be more
thoughtful or creative about what kinds of
comparisons might you be interested in doing. But that’s something in
this article search API. You can also get something
about word frequency. So there’s a lot
of possibilities of things you might explore. If you make stuff,
share them with me. I would love to hear about it. And I hope that this
was useful to you in how you might use
The New York Times API. This is me signing off. WFMXY YouTube ITP
hot lights room, weird place over the
window and a computer and– goodbye. I’ll talk to you. I don’t have a hashtag for you– #uncomfortablepersonDan. Goodbye. Oh wait, this mouse fell
asleep, sleeping mouse again.

37 comments

  1. Hello, may you explain the loadJSON and noCanvas methods please? When I type those into sublime, they do not highlight for me and when I run the code it does not work for me. I put this javascript code in script tags a the bottom of an html document.

  2. Hi the key you are using only returns 10 results per second i.e. the query ends up returning only 10 results, is their anyway to bypass this limit.

  3. very informative especially about little details. I watched all the videos in this particular series "working with data" . I feel much more confident now. Great job !

  4. Hi Daniel I really enjoy watching your video, btw is there any of your video explaining the other way to interact with a website that doesn't give us the API documentation? thanks!

  5. You are such an excellent teacher. I was terrified of all this stuff until I started watching your videos. Thanks a ton!

  6. when he said "by the way…, oh never mind", @time 4:17, I'm 90% sure he was about to say he adopted the word "rainbow" at Wordnik 😉 =) http://www.wordnik.com/words/rainbow how much do you love RAINBOWS!!!

  7. Great Stuff bro keep it up !!!. One question, How do you validate that the snippet of the H1 you created really belongs to that headline?

  8. I am wondering how to solve the problem that this API always returns just a few records of the data. Do I just narrow down by date_begin and date_end as much as I can so that I can get all records that are not over NYT's limitation?

  9. Great Tutorial. I have created a relevant website upon API http://fresh-news.net/. I want to know any suggestion about design and implementation of new functionality

  10. It doesn't work on 5p web editor 🙁 However, all the API urls works successfully from browser. but no responses from the consolle on the editor…why? Any ideas? Other suggestions/tutorials to do a similar request to a API?

  11. How can i add user input in here? Can someone point me to the right direction. I kind of get it and did those examples where we draw those humidity and temperature ellipses. But i'm not sure how to add that to the NY Times. Really enjoying Dan's teaching style.

  12. hey so this is my code for making a button to return articles headlines and their snippets upon querying…and it does so only once. When I enter a new word to search, it doesn't change the display. any ideas why would be awesome 🙂

    var url1 = "https://api.nytimes.com/svc/search/v2/articlesearch.json?q=";
    var url2 = "&api-key=syatA1onmp09w5lnOIp7LGa52fQwODDs";
    var wordData;
    var input;

    function setup()
    {
    //createCanvas(600,600);
    var button = select("#submit");
    button.mousePressed(wordAsk);
    input = select("#word")
    }

    function wordAsk()
    {
    var url = url1 + input.value() + url2;
    loadJSON(url,gotData)
    }

    function gotData(data)
    {
    wordData = data;
    }

    function draw()
    {

    if(wordData)
    {

    var articles = wordData.response.docs;
    for(var i = 0; i<11; i++)
    {
    createElement('h1',articles[i].headline.main);
    createP(articles[i].snippet);
    }
    }
    }

Leave a Reply

Your email address will not be published. Required fields are marked *