You’re not alone if you’ve been holding onto your favorite pair of “skinny” jeans that no longer fit and refuse to let them go, like a baby unwilling to part with her beloved security blanket.In fact, a British survey found that the average woman has 12 items of clothing worth $459 gathering dust in her closet because the clothes are either too big or too small for her to wear.But keeping clothes you can’t fit into—whether they’re the size 2 jeans from your 20s or the size 20 jeans you shed after losing weight two years ago—can do more harm than good.MORE: Restyle Your Old Outfits—For Free!Bryn Taylor, a personal stylist in New York City and founder of The Re-Stylist, knows all-too-well about women with dueling wardrobes since nearly all of her clients have clothes in their closets that they can no longer wear. “They either have one piece or half of a wardrobe that doesn’t fit,” says Taylor. “That’s very common. It depends on the person, but the majority of time we have to talk it out a bit—some clothes are not easy to part with. It’s so emotional.”That’s because parting with that pair of sexy, fitted black jeans or the slinky dress you used to slip on when you were hitting the town three to four nights a week (pre-responsible job or pre-baby) may mean saying goodbye to a former life. Or an “ideal” weight you always wished—and planned—to go back to.“Holding onto those clothes is often a reminder of this life you used to have that was cool,” explains YouBeauty Psychology Advisor Art Markman, Ph.D. “Those ‘skinny’ jeans or that really cute dress are reminders of when you used to be able to go out until 1am. Getting rid of those clothes is symbolic of ditching the life you had. And deep down you wish you could fit back into those jeans. But you have to recognize how different your life would have to be in order to fit back into those clothes.”The Lure of ‘Skinny’ ClothesBut not everyone hangs onto their too-tight clothes because they’re nostalgic for a magical weight they used to clock on a scale. “Some clients believe that by keeping their skinny clothes, they will serve as motivation to jumpstart their diet or workout routine, but that’s not going to do it,” says Taylor. “This pair of jeans isn’t going to get you out of bed and on the treadmill—it’s just a pair of jeans.”And keeping too-small clothes around can serve as a constant reminder that you’re not at that “perfect” weight (whether or not that weight is realistic for your body shape), which can take a toll on your self-esteem. “It could make them feel bad about their bodies,” notes Stephen Franzoi, Ph.D., professor of psychology at Marquette University in Milwaukee, WI.Fear of Giving ‘Fat’ Clothes AwayOn the flipside, letting go of “fat” clothes after shedding excess pounds can stir up other feelings, like you’re tempting fate by tossing them out. You may think, “If I get rid of these clothes and the weight comes back on, I’ll have nothing to wear!”  Adds Taylor, “I had a client who lost a significant amount a weight a year ago and she tells me she won’t get rid of her ‘fat’ clothes because she thinks of it as a jinx and that the weight will come back on.”Franzoi acknowledges that for people who have been overweight or obese for most of their lives and then lost a significant amount of weight, accepting their new slimmer selves and wardrobe—and letting go of those old clothes—is a big psychological adjustment.“They’re going to have to revise the way they think of themselves and that gets them out of their comfort zone,” explains Franzoi. “Even if they don’t like the way they are, there’s a predictability in their lives. If they take on this new identity, then they’re in a realm they’ve never been in before and that makes them insecure. Part of the time people hold onto a past self they didn’t like is because there’s security in that old self.”When to Toss Those ClothesSo how can you tell whether it’s time to toss the clothes that no longer fit, or hold onto them a little while longer and keep the dream alive (or the jinx at bay)?“Ideally, you should not have anything in your closet that doesn’t fit—that’s the goal,” says Taylor. “[Otherwise] you have that hanging in your closet, thinking, ‘I used to be able to fit into that.’ Stay in the moment of your current body shape.”QUIZ: What’s Your Body Type?But there are exceptions—and they don’t include swearing you’re starting your diet tomorrow or signing up for a single spinning class. If you’re pregnant or just had a baby or if you’re starting to train for a vigorous fitness activity, such as a marathon, your weight is going to fluctuate, so now’s not the time to get rid of clothes that don’t currently fit, notes Taylor.Also, there may be some good reasoning behind hanging onto your “fat” clothes if you’ve lost a lot of weight. “Look at the data—most people who lose weight tend to gain it back again because they haven’t changed their lifestyle,” notes Markman. “Holding onto these clothes may be a reflection or understanding of the fact that you’re probably not destined to stay the size you ended up.”That said, if you’ve kept the weight off for a year, it should be safe to donate your former “fat” clothes or even see if any of them are worth getting tailored to fit your body now. “If you have a more classic style, like trouser pants, then those are very easily altered,” says Taylor. “Your tailor can be your new best friend.”As for the rest, consider putting those unwearable clothes in a garment bag and out of sight (under the bed, on a high shelf or in the basement).It will save you time and stress when you’re scrambling to get dressed before work in the morning. “Your closet should be working for you 100 percent of the time,” says Taylor. “The last thing you need is to sift through your closet and say, ‘This one fits when I’m five pounds lighter.’ Everyone’s time is precious—it’s not worth it.”MORE: The Secrets to Loving Your BodyWhat’s more, continues Taylor, it’s not worth the toll those taunting clothes can take on your self-esteem. “If you open your closet and those ‘skinny’ or ‘fat’ clothes are staring at you, it’s going to weigh on you in some way—no pun intended.”