First off, warranty starts on the day the car is sold at the miles on the odometer at the time of delivery to the customer. However, the warrenty ends when the odometer hits 36,000 miles (bumper-to-bumper) or 36 months after the day it was sold. They do not add 36K miles to the starting mileage, it just plain ends at 36K. Therefor, you get less than 36K miles of warranty in reality, but you get the full time of warranty (assuming you run out of time before miles). The typical new car arrives from the factory with 2-8 miles on the odometer and is sold with 5-20 miles on the odometer.
We don't use one specific car for test drives, so there isn't a "test drive" car with tons of miles on it. Any car on our lot can be test driven, as each customer may want to try out a vehicle with different trim & equipment levels. A few test drive miles on a car is no big deal, and it is incredibly rare that someone would trash a new car on a test drive. We require a salesman to go on any test drive of a high performance vehicle to prevent thrashing on the hot cars. Spirited driving is OK, but tearing up tires & clutches is not.
Dealer trades are very common. We do about 70 of them monthly to get specific vehicles our customers want when we don't already have it, or to accomodate other dealers in the same position. We only trade within the Pacific NW and prefer closer dealers to distant dealers, but it is still common for dealer traded rigs to have 20-300 miles on the odometer. When I bought my 03 Mach 1 (brand new) we had to get it from out of state and it had 405 miles on the odometer at the time of purchase. Not really a big deal to me. If we trade in a vehicle for a customer, we always let them know it will have miles on it before we make the trip.
Most dealers that we trade vehicles with employee older, retired people to do their dealer trades, by the way. All 4 of our drivers are gray haired and they are accountable for any damage to the vehicles they bring back if it wasn't noted on the paperwork before they left the other dealership.
Dealer "demonstrators" (commonly called a demo) are getting rarer out there. Technically, they can be sold as new (and get the factory rebates) as long as the odometer doesn't go past 6,000 miles. Most dealership owners have a demo. Our owner is currently using a Lincoln MKS as his demo, for example. His last demo was a Ford Flex and it's on our lot for sale now at a sizable discount (has 1,885 miles on the odometer). Most demo's are discounted heavily to account for the miles. Demo's are typically loaded up models that are driven responsibly (consider that it's usually the dealership's owner using the vehicle), so they make pretty good buys if you get the right discount.
I hope this insider's look will help explain how miles "happen" on new vehicles.