Ammon
There's really not enough info here for even the roughest of guesswork. There are so many possible causes of changes in positions and it could just as easily be any of them. It may also be that the thing that saw your ranking go up is not the same thing that saw your ranking go down… Or it could be that both are part of the same thing, and caused by trending or freshness boosts somewhere at the search engine side.
SEO users all tend to be prone to a couple of well known biases. We tend to assume that any increase in ranking is because of what we did, and any decrease in ranking is because of something Google did. Our minds want to take all of the credit, and none of the blame. 😃
It may well be that your links ONLY had a negative effect and it was a change at Google that temporarily increased your rankings. But except where an SEO is a jaded old timer, used to watching and most importantly replicating a thing before being sure the two are cause and effect, the vast majority of SEO users see every upward twitch as proof of their great work.
It could be that one or more of the links you built were toxic and once Google had figured those links in, either your site lost some trust, or even that it allowed Google to identify other links that were not really genuine and discount those too.
Or it could be nothing at all to do with the links you got, and maybe other things about the site, or other backlinks, has changed. And, it could be nothing but the whole thing being Google temporarily tweaking the algo for that Search Engine Result Page (SERP) intent, both the temporary increase as the new math rolled through, and the eventual decrease, and those links did nothing at all. Believe it or not, with most link building done, that last is the most likely.
💟👍🏽4

Truslow » Ammon
Is spot on, above. So read that before this. I'm just going to provide some thoughts on how to avoid falling into the traps and confusion he's describing.
The first thing that many SEO users do that really screw them up is to look at the Search Engine Result Pages (SERPs) and just study where their sites/pages are within those results. This is a MAJOR mistake. You need to be familiar with the entire set of results. You need to know the other sites appearing above and below you – and what they are offering. You need to know what sorts of things are appearing as featured snippets and rich results. You need to be familiar with the set of things as a whole.
I call this the "Know The Neighborhood Method." Imagine for a moment that you're driving to work through a town. You're not all that familiar with the town but you know YOUR route through town and where every turn and traffic signal is. Now… one day, there is construction blocking your path and you've got to find another way to complete your journey to work and, dammit, your GPS is broken. Your first emotion is that the construction crew did it on purpose just to annoy you. Then you realize that's not actually the case, but you aren't really sure how to navigate the new landscape or where any of these offshoot roads are going to take you. It's going to take you a long time to get to work, today.
Now… imagine that you really KNOW this town well. You hit the construction and you know, "Okay… this changed, but if I turn here and take a left on Elm, I'll come out on the other side. No problem." You don't panic, you don't get angry, you just know this region like the back of your hand. You adapt, and proceed.
Knowing your neighborhood in SEO works very similarly. If you know ONLY your place in the results – and then you are forced to change your place, you can't see how or even why you're going to get out of it. If you know the whole neighborhood, though – you can more easily see what's in the way, what changed, and how to get around it and adapt your plan to find the best route to your goal.
As Ammon points out – this might have to do with your links, but maybe not. (I'd go so far as to say "probably not" unless there's something sketchy about the way you've been obtaining links). There are LOTS of things that could be in play here – but it'll be hard to spot exactly what is going on with the results if you don't know what they looked like last week as compared to how they look now. Maybe your page is now featured in a "People Also Ask" (or some other) snippet. This often comes at the cost of having your organic link pushed back a ways because it's already represented on the first page. But your rank tracking doesn't give credit for your ranking in those – so it just appears to be a loss even though that scenario would mean a gain from page 2 to page 1.
Know Your Neighborhood – like the back of your hand. And if you do, these things will be easier to figure out exactly what happened, what is in your way, and how to navigate around the problem (if it is a problem at all).
(True Story: I had a person upset that, according to their rank tracking tools, their #1 keyword dropped from the #1 spot to somewhere on page two. I wasn't familiar with that neighborhood since they weren't yet a client, but I decided to take a look. Turns out that their organic ranking HAD dropped to the #12 spot for me – but they also owned the huge featured snippet at the top with their brand, an incomplete list that encouraged a click to see more, and a branded photo of the item in question. They called me panicking because they'd lost something, completely unaware that they'd accidentally landed the White Whale of modern SEO.)

📰👈

🔗🏹