The reference is that there is no reference. When you are not required to do something, you won't find a reference saying it's not required. The absence of any reference requiring you to do it is your reference.
There are, of course exceptions. On the initial call (such as changing control sectors or frequencies, you should include your actions such as descending, and the altitude you're vacating. A proper courtesy would be XXX Center, XXXY Five thousand, climbing one zero thousand. This includes all the relevant information.
The AIM provides guidance as to what you should do, vs. what you are required to do. It is not regulatory, but represents a collation of regulatory items and material from advisory circulars and and other official publications.
AIM5-3-1
2. The following phraseology should be utilized by pilots for establishing contact with the designated facility:
(a) When operating in a radar environment: On initial contact, the pilot should inform the controller of the aircraft's assigned altitude preceded by the words "level," or "climbing to," or "descending to," as appropriate; and the aircraft's present vacating altitude, if applicable.
Skipping ahead, we visit AIM 5-3-3, which provides additional reports in addition to the initial call up specified above, and position reporting in a non-radar environment. Again, frequency congestion determines w(h)eather these calls should be made. Tying up an already busy frequency with a courtesy call that you're vacating an assigned altitude when you're in radar contact and the controller can see you vacating the altitude is wasteful, discourteous, and a safety hazard.
AIM 5-3-3. Additional Reports
a. The following reports should be made to ATC or FSS facilities without a specific ATC request:
1. At all times.
(a) When vacating any previously assigned altitude or flight level for a newly assigned altitude or flight level.
(b) When an altitude change will be made if operating on a clearance specifying VFR-on-top.
(c) When unable to climb/descend at a rate of a least 500 feet per minute.
(d) When approach has been missed. (Request clearance for specific action; i.e., to alternative airport, another approach, etc.)
(e) Change in the average true airspeed (at cruising altitude) when it varies by 5 percent or 10 knots (whichever is greater) from that filed in the flight plan.
(f) The time and altitude or flight level upon reaching a holding fix or point to which cleared.
(g) When leaving any assigned holding fix or point.
This same laundry list will be found in Chapter 10 of the Instrument Flying Handbook (FAA-H-8083-15).
ATC may predicate altitude assignments under some circumstances upon reports of vacating an altitude. In such cases, timely and accurate reporting is important.