It seems that "idling-up" function doesn't operate properly. Possible culprit would be one of these.
1. AC switch (It tells status of the AC compressor =ON or OFF= to ECU.)
2. AAC valve (It controls idling speed by the output from ECU.)
Anyway, I think the AAC valve shouldn't be removed as long as your car's engine management is in original condition.