Tough question - so many facets.
I grew up in Victoria, now live in Vancouver have for about 15 years, however I still have significant ties to Victoria so am over quite a bit. Keep in mind all of this is generalization so:
As Toronto is to New York Vancouver is to Toronto as Victoria is to Vancouver.
This applies to just about everything and is either good or bad depending on your perspective.
Victoria is a very small town if you are coming from a big city. It is very WASP in comparison to either Vancouver or Toronto as someone else already said. Victoria has a fairly vibrant music and quite a large artistic community. However the base population is quite small so in comparison to a larger city it will feel quite limited. Vancouver in contrast has just about everything in terms of culture, great music artists, etc. however not as much as TO (as I understand it) again TO has a much larger population base to support more variety. Think New York - Toronto.
Work is quite difficult to come by (outside of the tourist service industry) in Victoria in general. This is a retirement/ government/tourist town. These three industries drive everything and lots of people with lots of experience want to semi retire in Victoria so you are competing with those people for the good jobs. I don't know the actual employment market in Victoria right now so I may be wrong, however this is the historical pattern. In general Victoria pays 20% or more less for the same jobs than in Vancouver and the cost of living is not 20% less. This does not apply to Union/government jobs but for pretty much everything else it does. In my field of work the difference is more like 50% less. Moving to Vancouver more than doubled my annual earnings.
In Vancouver things are hopping right now - if you were a conveyancing secretary, or paralegal you could find a job within a day of arriving easily. (This is the world I work in so I am familiar with it - I have people asking me regularly if I know anyone.) Getting ready for the Olympics and the boom in resource industries is driving the economy into overdrive here.
Work, and pretty much everything else happens at a slower pace in Victoria. A much more relaxed city. Vancouver is much busier and things happen at a much faster pace. Toronto is busier and faster again, New York faster again.
Renting is shifting quite a bit in both markets, property values have changed considerably in the past year so there has been a lot of movement. The rental market is in flux at the moment. $600 - 700 will get you a reasonable 1 bedroom in Victoria I believe. I rent out two one bedroom suites in Victoria for considerably less than that, but my tenants have been with me for years. Don't know the Vancouver rental market.
Utilities will be cheaper in both cities. No need to spend so much on heat. You cannot live without a car in Victoria. You can in Vancouver, but not easily. To do it you would pretty much have to live downtown.
Health care is an odd question as basic medical care is pretty much the same across Canada. If you have specific concerns you should probably talk to a medical professional as there are significant differences in what the BC Medical pays compared to OHIP. Some things are better some not as good. Vancouver has a very good research/teaching hospital at UBC and a very good Children's Hospital. Victoria has two very good hospitals (I lived with a Nuc Med Tech for a couple of years who worked in both and is still there). Both have good reputations.
Two final points. First, Victoria is on an Island, and while you are only 50 or so miles from Vancouver and a little more from Seattle, in practical terms you are four hours from both. I can almost get to Hawaii faster than I can get to Victoria. In the winter the ferry runs every two hours and if you want to drive you line up an hour early or pay extra for a reservation. It can be psychologically isolating to live on an Island - not logical but true. I go to Victoria 3 or four times a year. When I lived in Victoria I would go to Vancouver maybe once a year.
Second both cities get significant rain and cloud. Vancouver more than Victoria. If you are subject to winter depression it can be a very difficult city to live in. I bring this up as you are moving largely as you are fed up with the weather. It is much warmer here and you don't have to deal with the snow. However it rains - alot.
Hope that helps