what’s going on everybody welcome to part 13 of the go language tutorial
part 13 of the go language tutorial
part 13 of the go language tutorial series in this tutorial what we’re doing
series in this tutorial what we’re doing
series in this tutorial what we’re doing is just building on the last tutorial
is just building on the last tutorial
is just building on the last tutorial with our news aggregator web app idea in
with our news aggregator web app idea in
with our news aggregator web app idea in the previous tutorial we were talking
the previous tutorial we were talking
the previous tutorial we were talking about looping and how we could loop over
about looping and how we could loop over
about looping and how we could loop over this the information that we’re pulling
this the information that we’re pulling
this the information that we’re pulling which in this case is just a bunch of
which in this case is just a bunch of
which in this case is just a bunch of sitemaps now what we want to do is
sitemaps now what we want to do is
sitemaps now what we want to do is actually visit those site maps and then
actually visit those site maps and then
actually visit those site maps and then pull the information from there which is
pull the information from there which is
pull the information from there which is actually the load you know the actual
actually the load you know the actual
actually the load you know the actual articles themselves their titles and
articles themselves their titles and
articles themselves their titles and keywords stuff like that so that’s what
keywords stuff like that so that’s what
keywords stuff like that so that’s what we’re gonna be doing here we also need
we’re gonna be doing here we also need
we’re gonna be doing here we also need to clean this up a little bit I kind of
to clean this up a little bit I kind of
to clean this up a little bit I kind of was really explicit with every part that
was really explicit with every part that
was really explicit with every part that we wanted to break down here but what we
we wanted to break down here but what we
we wanted to break down here but what we can do is actually we can we can
can do is actually we can we can
can do is actually we can we can simplify this quite a bit so it’s all
simplify this quite a bit so it’s all
simplify this quite a bit so it’s all just one struct which is gonna make
just one struct which is gonna make
just one struct which is gonna make sense you wouldn’t want to I mean you
sense you wouldn’t want to I mean you
sense you wouldn’t want to I mean you can break them down is this far if you
can break them down is this far if you
can break them down is this far if you really wanted to the problem is as this
really wanted to the problem is as this
really wanted to the problem is as this program grows that’s going to get super
program grows that’s going to get super
program grows that’s going to get super problematic like well like to do the
problematic like well like to do the
problematic like well like to do the next the next site map and all the
next the next site map and all the
next the next site map and all the information that we want from there
information that we want from there
information that we want from there which will be titles keywords and the
which will be titles keywords and the
which will be titles keywords and the location you’re gonna end up with like
location you’re gonna end up with like
location you’re gonna end up with like 15 structs if you’re not careful so so
15 structs if you’re not careful so so
15 structs if you’re not careful so so let’s go ahead and clean this up at
let’s go ahead and clean this up at
let’s go ahead and clean this up at least a little bit so one of the things
least a little bit so one of the things
least a little bit so one of the things that we could do right out of the gate
that we could do right out of the gate
that we could do right out of the gate is like if you’ll notice here sitemap
is like if you’ll notice here sitemap
is like if you’ll notice here sitemap index is a you know it’s luke has just
index is a you know it’s luke has just
index is a you know it’s luke has just one value in it which is locations which
one value in it which is locations which
one value in it which is locations which is a slice of the location type well if
is a slice of the location type well if
is a slice of the location type well if we look at the location type it turns
we look at the location type it turns
we look at the location type it turns out that’s just a string type and so all
out that’s just a string type and so all
out that’s just a string type and so all we really need to do at this point would
we really need to do at this point would
we really need to do at this point would be to take rather than have locations be
be to take rather than have locations be
be to take rather than have locations be a location type we can actually just
a location type we can actually just
a location type we can actually just make that a string type and that solves
make that a string type and that solves
make that a string type and that solves this problem here well to an extent we
this problem here well to an extent we
this problem here well to an extent we also need to reference the tag itself so
also need to reference the tag itself so
also need to reference the tag itself so its location inside of the sitemap tag
its location inside of the sitemap tag
its location inside of the sitemap tag so to handle for that you just do a
so to handle for that you just do a
so to handle for that you just do a greater than sign and go to the next tag
greater than sign and go to the next tag
greater than sign and go to the next tag so this means the location tag
so this means the location tag
so this means the location tag underneath the sitemap tag so so now if
underneath the sitemap tag so so now if
underneath the sitemap tag so so now if you just make that slight change we can
you just make that slight change we can
you just make that slight change we can actually get rid of both the location
actually get rid of both the location
actually get rid of both the location struct but we can also get rid of this
struct but we can also get rid of this
struct but we can also get rid of this string method
string method
string method we won’t actually need that anymore
we won’t actually need that anymore
we won’t actually need that anymore because now it’s a string type as we
because now it’s a string type as we
because now it’s a string type as we iterate over locations so that
iterate over locations so that
iterate over locations so that definitely simplifies things quite a bit
definitely simplifies things quite a bit
definitely simplifies things quite a bit for us so now we want to talk a little
for us so now we want to talk a little
for us so now we want to talk a little bit about what happens when we go and
bit about what happens when we go and
bit about what happens when we go and visit one of the site maps from here so
visit one of the site maps from here so
visit one of the site maps from here so let me go ahead and just pull one up
let me go ahead and just pull one up
let me go ahead and just pull one up real quick just so we can kind of see
real quick just so we can kind of see
real quick just so we can kind of see what we’re working with
what we’re working with
what we’re working with so I’ll just pull up the politics
so I’ll just pull up the politics
so I’ll just pull up the politics sitemap and I’ll just bring that over so
sitemap and I’ll just bring that over so
sitemap and I’ll just bring that over so here is that let me just make it fit so
here is that let me just make it fit so
here is that let me just make it fit so as you can see here this time everything
as you can see here this time everything
as you can see here this time everything is contained in the the most parent tag
is contained in the the most parent tag
is contained in the the most parent tag is URL set and then inside of URL set
is URL set and then inside of URL set
is URL set and then inside of URL set you’ve got URL inside of the URL tag we
you’ve got URL inside of the URL tag we
you’ve got URL inside of the URL tag we have the location we’ve got n colon news
have the location we’ve got n colon news
have the location we’ve got n colon news the publication you’ve got title
the publication you’ve got title
the publication you’ve got title keywords all that and the n colon news
keywords all that and the n colon news
keywords all that and the n colon news is something I’m I don’t really see too
is something I’m I don’t really see too
is something I’m I don’t really see too too much I’m guessing they’re just
too much I’m guessing they’re just
too much I’m guessing they’re just calling this the like why they’re using
calling this the like why they’re using
calling this the like why they’re using n colon I don’t really know something’s
n colon I don’t really know something’s
n colon I don’t really know something’s like super familiar with XML and you
like super familiar with XML and you
like super familiar with XML and you want to let us know why they do n colon
want to let us know why they do n colon
want to let us know why they do n colon rather than just news because basically
rather than just news because basically
rather than just news because basically this is still a news tag it’s just like
this is still a news tag it’s just like
this is still a news tag it’s just like a news tag of of n which I guess is you
a news tag of of n which I guess is you
a news tag of of n which I guess is you know new I’m guessing is news but it
know new I’m guessing is news but it
know new I’m guessing is news but it makes no sense to me why didn’t you just
makes no sense to me why didn’t you just
makes no sense to me why didn’t you just call the tag news so if anybody can
call the tag news so if anybody can
call the tag news so if anybody can answer that question let me know anybody
answer that question let me know anybody
answer that question let me know anybody from Washington Post watching anyway
from Washington Post watching anyway
from Washington Post watching anyway [Music]
[Music]
[Music] wondering where all their new traffic is
wondering where all their new traffic is
wondering where all their new traffic is coming from and moving along so anyways
coming from and moving along so anyways
coming from and moving along so anyways this is what we want to parse so the
this is what we want to parse so the
this is what we want to parse so the structure is a little different so just
structure is a little different so just
structure is a little different so just again just you can pull this up the you
again just you can pull this up the you
again just you can pull this up the you can go to the text-based version I have
can go to the text-based version I have
can go to the text-based version I have a sample of this so if things do change
a sample of this so if things do change
a sample of this so if things do change you’re in luck I got a sample just like
you’re in luck I got a sample just like
you’re in luck I got a sample just like I did before that you could still work
I did before that you could still work
I did before that you could still work with or you can adapt this to your own
with or you can adapt this to your own
with or you can adapt this to your own but basically we know we’re gonna have
but basically we know we’re gonna have
but basically we know we’re gonna have to go inside the URL set inside the URL
to go inside the URL set inside the URL
to go inside the URL set inside the URL one of the things we are interested in
one of the things we are interested in
one of the things we are interested in is the location tag but we’re also gonna
is the location tag but we’re also gonna
is the location tag but we’re also gonna be interested in from inside URL inside
be interested in from inside URL inside
be interested in from inside URL inside the news tag we also want title in
the news tag we also want title in
the news tag we also want title in probably key words so quite a bit more
probably key words so quite a bit more
probably key words so quite a bit more structure that we want to pull in and
structure that we want to pull in and
structure that we want to pull in and that’s why we want to really get our
that’s why we want to really get our
that’s why we want to really get our struck
struck
struck perfect because otherwise we’re gonna
perfect because otherwise we’re gonna
perfect because otherwise we’re gonna like I was saying before end up with
like I was saying before end up with
like I was saying before end up with like 30 struts probably more like like
like 30 struts probably more like like
like 30 struts probably more like like 10 struts or something like that but it
10 struts or something like that but it
10 struts or something like that but it would be kind of absurd so so that’s our
would be kind of absurd so so that’s our
would be kind of absurd so so that’s our sitemap index struct let’s go ahead and
sitemap index struct let’s go ahead and
sitemap index struct let’s go ahead and build a new struct and that one are
build a new struct and that one are
build a new struct and that one are basically a new type and that’s going to
basically a new type and that’s going to
basically a new type and that’s going to be type news and of course it’ll be a
be type news and of course it’ll be a
be type news and of course it’ll be a struct and then inside news this one’s
struct and then inside news this one’s
struct and then inside news this one’s going to have titles which will be a
going to have titles which will be a
going to have titles which will be a slice in the string form and the
slice in the string form and the
slice in the string form and the location of this utility here is gonna
location of this utility here is gonna
location of this utility here is gonna be it’s gonna be XML tags and then like
be it’s gonna be XML tags and then like
be it’s gonna be XML tags and then like we were discussing it’s the URL and then
we were discussing it’s the URL and then
we were discussing it’s the URL and then it’s within remember it was in colon
it’s within remember it was in colon
it’s within remember it was in colon news but for some reason I’m not even
news but for some reason I’m not even
news but for some reason I’m not even sure about the official name for
sure about the official name for
sure about the official name for something like this would be but you
something like this would be but you
something like this would be but you don’t use that you’re just gonna say
don’t use that you’re just gonna say
don’t use that you’re just gonna say okay it’s in the news tag and then it’s
okay it’s in the news tag and then it’s
okay it’s in the news tag and then it’s inside I’m sorry we are going to do
inside I’m sorry we are going to do
inside I’m sorry we are going to do keywords but title and I think title was
keywords but title and I think title was
keywords but title and I think title was under case let me confirm yeah okay yeah
under case let me confirm yeah okay yeah
under case let me confirm yeah okay yeah it looks like it should be under cased
it looks like it should be under cased
it looks like it should be under cased or a lower case rather okay so we have
or a lower case rather okay so we have
or a lower case rather okay so we have titles now we do want alright now we
titles now we do want alright now we
titles now we do want alright now we want to do keywords so keywords of the
want to do keywords so keywords of the
want to do keywords so keywords of the string form and again XML and then this
string form and again XML and then this
string form and again XML and then this one basically the same path actually
one basically the same path actually
one basically the same path actually it’s just rather than title it was
it’s just rather than title it was
it’s just rather than title it was keywords so keywords and then finally we
keywords so keywords and then finally we
keywords so keywords and then finally we have the actual locations so locations
have the actual locations so locations
have the actual locations so locations don’t forget capital L locations so it
don’t forget capital L locations so it
don’t forget capital L locations so it does get exported string and then this
does get exported string and then this
does get exported string and then this one stop it this one is located under
one stop it this one is located under
one stop it this one is located under XML and then this one was like it was
XML and then this one was like it was
XML and then this one was like it was just URL and then the location and just
just URL and then the location and just
just URL and then the location and just remember like one of the like location
remember like one of the like location
remember like one of the like location was at the same level as News was it’s
was at the same level as News was it’s
was at the same level as News was it’s just news content happens to contain
just news content happens to contain
just news content happens to contain other other tags that we were also
other other tags that we were also
other other tags that we were also interested in so that’s our news struct
interested in so that’s our news struct
interested in so that’s our news struct now let’s go ahead and go into our main
now let’s go ahead and go into our main
now let’s go ahead and go into our main function here now and basically
function here now and basically
function here now and basically everything here is is good for us
everything here is is good for us
everything here is is good for us the only difference is basically we want
the only difference is basically we want
the only difference is basically we want to visit
to visit
to visit now rather than just print out the
now rather than just print out the
now rather than just print out the location we actually want to visit that
location we actually want to visit that
location we actually want to visit that location instead so to do that we would
location instead so to do that we would
location instead so to do that we would just basically do the exact same thing
just basically do the exact same thing
just basically do the exact same thing that we’ve done you know up here right
that we’ve done you know up here right
that we’ve done you know up here right so so what we’re gonna do is where we’ve
so so what we’re gonna do is where we’ve
so so what we’re gonna do is where we’ve defined and probably probably you should
defined and probably probably you should
defined and probably probably you should define your bars like at the top here so
define your bars like at the top here so
define your bars like at the top here so that’s going to do that and then also
that’s going to do that and then also
that’s going to do that and then also bar in news and then so this will
bar in news and then so this will
bar in news and then so this will [Music]
[Music]
[Music] unmarshal into s then we start iterating
unmarshal into s then we start iterating
unmarshal into s then we start iterating through and then basically we’re just
through and then basically we’re just
through and then basically we’re just going to do the exact same thing here so
going to do the exact same thing here so
going to do the exact same thing here so I’m actually just gonna copy actually
I’m actually just gonna copy actually
I’m actually just gonna copy actually let’s just copy all three of these lines
let’s just copy all three of these lines
let’s just copy all three of these lines because we need to do all three of those
because we need to do all three of those
because we need to do all three of those things so just paste that in and then
things so just paste that in and then
things so just paste that in and then now rather than HTTP GET that means
now rather than HTTP GET that means
now rather than HTTP GET that means sitemap what we’re gonna do is get the
sitemap what we’re gonna do is get the
sitemap what we’re gonna do is get the the location whatever that happens to be
the location whatever that happens to be
the location whatever that happens to be as we iterate around bytes are gonna
as we iterate around bytes are gonna
as we iterate around bytes are gonna stay the same and then this time rather
stay the same and then this time rather
stay the same and then this time rather than on marshaling into the sitemap
than on marshaling into the sitemap
than on marshaling into the sitemap index we’re gonna unmarshal into our
index we’re gonna unmarshal into our
index we’re gonna unmarshal into our news struct so now we should wind up
news struct so now we should wind up
news struct so now we should wind up with pretty much all that same
with pretty much all that same
with pretty much all that same information that we have the only issue
information that we have the only issue
information that we have the only issue now is how we’re gonna start going
now is how we’re gonna start going
now is how we’re gonna start going through that information because at this
through that information because at this
through that information because at this point like when we just had locations
point like when we just had locations
point like when we just had locations okay that was just one value we just
okay that was just one value we just
okay that was just one value we just loop through that value and that’s all
loop through that value and that’s all
loop through that value and that’s all we really wanted to do but now in each
we really wanted to do but now in each
we really wanted to do but now in each of these sitemaps what we’re getting is
of these sitemaps what we’re getting is
of these sitemaps what we’re getting is title location and keywords well what do
title location and keywords well what do
title location and keywords well what do we actually want to do with that we want
we actually want to do with that we want
we actually want to do with that we want to display that on our web app and so at
to display that on our web app and so at
to display that on our web app and so at least to me my desire at this point is I
least to me my desire at this point is I
least to me my desire at this point is I want to put this into some sort of key
want to put this into some sort of key
want to put this into some sort of key and values system so in you know in
and values system so in you know in
and values system so in you know in Python that would be a dictionary in
Python that would be a dictionary in
Python that would be a dictionary in Django that’s gonna be a map so in the
Django that’s gonna be a map so in the
Django that’s gonna be a map so in the next tutorial what we’re gonna be
next tutorial what we’re gonna be
next tutorial what we’re gonna be talking about is how we can you know
talking about is how we can you know
talking about is how we can you know iterate through this and build a map
iterate through this and build a map
iterate through this and build a map that we can then pass basically to our
that we can then pass basically to our
that we can then pass basically to our our web app and then we’ll use that to
our web app and then we’ll use that to
our web app and then we’ll use that to display all the information that we want
display all the information that we want
display all the information that we want to show on the page so anyways that’s
to show on the page so anyways that’s
to show on the page so anyways that’s what we’re gonna be doing in the next
what we’re gonna be doing in the next
what we’re gonna be doing in the next tutorial if you have any questions
tutorial if you have any questions
tutorial if you have any questions comments concerns whatever up to this
comments concerns whatever up to this
comments concerns whatever up to this point feel free to leavin below
point feel free to leavin below
point feel free to leavin below otherwise I will see you in the next
otherwise I will see you in the next
otherwise I will see you in the next toriel
”
Be First to Comment