Libraries to render vector-based maps in the browser
Update: since posting this, Mapbox released Mapbox GL JS, which does WebGL-based vector basemap rendering in browser.
At this point, there's no established option. The only fully-vector in-production map out there is Google's MapsGL, and that's very limited due to browser support & performance.
The closest you'll get with open source at the moment is Kothic.js and rolling your own tiles with Kothic's script.
For just drawing vectors, Leaflet or OpenLayers will do just fine. Nothing new there.
For vector tiled maps (i.e different resolutions/levels of detail at different zoom levels), TileStache is a simple and good server. I have been using with lots of experiments with Polymaps (AFAIK, the only general-purpose working js tiled vector viewer). Some people have used OpenLayers for adding tiled vector support, but AFAIK, it has not been integrated to trunk in a year, so it probably won't in the near future for lack of interest.
TileStache (the server) itself is working very well. Nevertheless, I have found several bugs with Polymaps and since it is not being updated anymore (abandoned project), I cannot convince myself to use it in production (I'd have to maintain it myself... I can't justify it to myself just yet.)
My solution for me has been to write a custom OpenGL native vector renderer which is used in AmigoCloud. In fact, that is how all maps are rendered in our service. For that purpose, TileStache has delivered quite well.
You want to look at the TileStache Vector Provider. For examples on how to use it, you can look at a very basic vector test suite I wrote.
Sadly, I don't think Leaflet has support for tiled vectors.
If I understand your question correctly then both OpenLayers and Leaflet will load/show vector data.
For example
http://leafletjs.com/examples/geojson/
this loads a GeoJSON file onto the leaflet map which uses OSM via CloudMade in the background.
OpenLayers probably supports more different types of vector formats, like KML
http://dev.openlayers.org/examples/dynamic-text-layer.html
If you mean you are after a server to serve vector data onto the map then look at MapServer, GeoServer and ESRI ArcGIS Server which all will server vector data onto your map as a Web Feature Server (WFS).