Portfolios belong to whoever the agreement you signed to undertake the award state they belong to (that's why some form of learning agreement is a good idea).
Guidance states that portfolios should be the property of the candidate but that candidate should be prepared to return them for quaility assurance purposes (this makes centres nervous as it is hard to get them back). Effectively portofolios can be returned to candidates at any time BUT Centres generally return them as soon as they are complete and a successful EV visit has taken place for the relevant award.
Centres do need to keep unit summary sheets and candidate assessment records for 3 years. It may be the college has an overriding policy of retention but more likely it's a lazy solution to meet their requirements and does not help you - you should challenge it - awarding body guidance is on your side
