Well the heaviest part of the space elevator is its counterweight, mass about 100 tons in this estimate The physics of the space elevator - so not huge. Not enough to make any noticeable change in Earth's centre of gravity.
Most of the effect comes from launches, when you release spacecraft from the top of the space elevator into orbit or into interplanetary trajectories..
If you keep launching huge masses into outer space - the energy for those launches has to come from somewhere, and each launch gains energy from the Earth's rotation, so slows the Earth's rotation down.
It's the other way around when you land masses from outer space with the elevator. Each time you do that, you speed the Earth up, slightly.
So if it was a concern, you could balance things by making sure your launches and landings balance the budget of the Earth's angular momentum.
However, someone on stackexchange has done the calc. already. According to their calc, about 10^14 tons, 100,000,000,000,000 tons - or about a billion US aircraft carriers in mass launched on the space elevator would slow the Earth's spin down by 1 second.
Earth's mass is 10^21 tons approx.
So, I think no need to be worried about this for now :)