How to set up a QA dept.?
If you are going with Scrum, I suppose you mean all agile approach. Won't be Test Plans, bug reports and stuff like that a bit overhead?
As for the tools I would suggest to add wiki or jira as place to store tasks, associated user stories, and associated bugs.
As for setting up QA department (one person as department O.o ), I would suggest to drop 'QA dept.' label, just hire one more team member that will focus on testing. You may want person that worked in agile and has testing experience. It would be good if that person would know exploratory testing (tools&techniques).
If you think about test automation (since you already have Hudson) tester would need to have some skills in programming. On the other hand you could leave automation to developers and focus on getting good tester than average tester that can do some code. Anyway I would implement a bit of test automation, some regression.
One thing, don't fall into QA/process/test plans/documentation heavy/detailed manual scripts/process oriented thing. Keep it all agile, otherwise soon you will notice that it doesn't really work.
=================edit to address Jacko comment============================
So I assume, that not intend to contribute to this open source project, but rather make some plugins that will use its code but also extend its functionality, so it would suit your need in your own project? Cool.
So you have:
1) Open Source project developed by other party, with it's own schedule, project plan, etc
2) Extensions/plugins you develop, that make that project usable for you
3) Your own project that has some functionality delegated via plugins to opensource project.
I guess that all this integrates on code level, via some messaging mechanism. In this case I would say, that you need someone who can code, regardless of his background (although developer will be easier to find, but will he be interested in writing automation tests?).
Anyway you need automation tests for:
A) Obviously your project - write unit tests as you do now, or maybe a bit more.
B) Unit tests that will assure that any change in your main project doesn't break its interaction with plugins/extensions (those for opensource project)
C) unit tests of plugins/extensions - it is code you write, so you need unit tests as for you need A for your project
D) (not so obvious part) you need tests to see if changes in open source project affect your plugins.
Of course A and B as C and D will overlap somehow, but formally that's what you need to be aware of.
So, getting back to original question, I don't think that you need QA department (by the way, ain't scrum saying to drop department labels and have one team?) in traditional meaning. Get a guy that can (and likes) to create automation tests, possibly on Unit Test level. Go with Scrum. Have one team. Skip comprehensive documentation, formal process, ISO/IEEE standards for testing. Create lightweight documentation, just to have main/general objectives/approaches/assumptions to refer to.
...and if it doesn't work, discuss it on retrospective, try to tweak things, try something new. Continuous improvement!
You indicate that you want a QA group/department, but you don't say why. There's a lot of assumptions around what a QA group does.
The base nature of QA is to insure the right product is built right, the first time. Software Quality Assurance is normally a focus on the process of developing code.
Or, are you just looking for somebody else to test your code for you ?
If your focus is on the testing aspects of QA, I would push as much as possible into the development portion of your group. Automated tests will go a long way to help improve the chances of getting it right the first time. Additionally, tests significantly decrease the risk of changes within the product. And, don't divert the majority of the test writing onto a subset of the team. You want to make testability a key aspect of your design and developers that don't write enough tests, don't weight it sufficiently in their designs. Counting on a human or set of humans to run through a bunch of tests means you can't run them very often and you'll tend to underemphasize the older regression tests.
If your focus is on QA and not just testing, it is worth assigning somebody who's primary role is to look at the product, its complexity, metrics and test coverage to determine if there are enough tests in the right places in the code base. This role would be no different than an architect monitoring the overall design or somebody specialized in usability and/or design guiding the user interface development. The role should be to champion the right work, not necessarily to implement the work.
The one exception I personally have with all of the above is the need for at least one person with the right mindset to perform exploratory testing. Exploratory testing should be a subset of all the testing described above. And, as problems are found, should be used to feedback into the automated testing (eg if a bug is found, part of the act of fixing should be to write test that demonstrates the bug and the fix. And, if the test coverage is low for the code complexity, write several other tests to makes sure there aren't additional bugs). UIs, in particular, need exploratory testing. The permutations of usable ways to accomplish a task, as well as the difficulty of automating tests in older UI frameworks, are often well served by exploratory testing.
Update: You might want to check out Episode 164: Agile Testing with Lisa Crispin