Is it self-plagiarism to reuse sources?
In general, no (and how did you get access to your school's plagiarism checker)
If the papers are related they should share sources. From a published peer-reviewed standpoint, I'd bet several papers reuse half their sources.
This sounds like a paper for a grade though. In which case you actually have a 3rd option. Talk to the professor or TA. Bring the first paper and this paper with you and let them make a decision. Assuming you didn't plagiarize, they'll see the content is different.
Remember, part of taking a university course is to learn how to write papers with novel ideas. The professor and TAs are there to help you learn how to present your ideas without plagiarizing sources. Especially coming before the paper is due shows a good-faith effort.
Self plagiarism is reusing your old work and representing it as new. You avoid it by citing the old work within the new, just as you would the work of any other author.
If I read your question correctly, you copied something like the bibliography of the old work, and maybe a section on "prior work" into the new paper from the old. The checker looked for identical sections of text and found these.
I don't actually see much of an issue with that, actually. There is a principle that applies and it shows up a lot in math and sometimes in CS. Often there is only one way to correctly say something. In math, the definition of the derivative is pretty standardized, for example, so every book and paper that repeats it has essentially the same text. A bibliography is like that. It uses a standard format and ordering and the items in it have a standard form. So two papers on the same topic will have large overlapping areas of the bibliography. That doesn't mean they are plagiarized.
A section on prior work might be a bit more problematic. You can avoid a charge of self plagiarism if you just state in the new paper that the prior work is the same as for the previous one and is repeated here only for the reader's convenience. Since there is a citation to the old work, there is no self plagiarism. But an automated plagiarism checker will probably still flag it.
The reason to avoid self plagiarism is that a reader of the new work wants a complete context of the ideas presented. The old paper establishes a certain relevant context with its citations and bibliography and if the new work doesn't point to the old one then the reader might miss that important context. So, we cite, letting the reader go back for more context as needed. But nothing you describe here seems to indicate an issue with this, as long as your new paper cites the old one.
Keep all of this in mind and be prepared to use it if you go see your professor as sevensevens suggests. At the most, you might have to make some minor changes to the text and add a citation or two for clarity. But make it the professor, not the TA, you speak to. Oddly, the professor will probably be less pedantic about such things and is the final authority over what you are permitted.
A final note: Often rephrasing something you wrote earlier to avoid automated checking is worse than just copying it and giving a citation. This is because it can seem to imply an intent to disguise. Copying large sections is probably not a good practice, but if it makes things clearer to a reader it may be better for shorter passages. But give the citation and state that the section is "repeated for convenience of the reader". Then it is clear.