Speed should not be an issue as long as you have enough memory and cpu to support the number of users you usually have. Flash and other video can be resource intensive on the server depending on how you are virtualizing. Another speed issue is how the two libraries are networked. If you have limited bandwidth between the two branches such as a T1 then you'd likely want a server at each of the libraries so that connections are local within the library.
Secondly you'll want to think about how your sessions are cleared out between patrons. I'm not sure how or if you are doing it currently but the method may not work virtualized. I'm not overly familiar with Hyper-V so this may be mute. On this vein is how long it takes for the virtual session or thin client to boot/reimage or generally become available for use. I've seen some poorly constructed systems (full or thin client) in some academic libraries that take 5 minutes to become usable due to how they boot and image.
One thing that has worked well for us with thin clients is mounting them with a VESA mount out of reach from patrons and then mounting a FrontX modular port onto the desk instead. While the case is meant to go into a 3.5in bay in a computer we find it looked fine screwed under our desks. This means patrons are actively using a $10 part which is fairly sturdy and cheap to replace, rather than a direct port on a thin client that would mean eventually replacing a thin client. In the 4ish years we've had 100+ thin clients out we've only had to replace two due to them frying (power here is not good and surge protectors can only do so much).
Here is the frontX site: http://www.frontx.com/