At first, I'd assumed that the winter plumage of the lesser goldfinches was what made the American goldfinches more dull than the brilliant canary yellow they prominently display in spring and summer (first mistake, never make assumptions - lesser goldfinches do not molt so boldly). What should have clued me in are the white wing bars, which are far more prominent and clearly defined than any markings on the lesser goldfinches. Yet just recently, I spotted what is unmistakably a male lesser goldfinch, complete with his brighter plumage and scruffy black cap, feeding alongside the muted finches. As they were feeding beneath my niger feeders (and you wouldn't believe the mess I'm going to have to clean up in the spring), I noted a very discernible size difference between the two species. A bit of cross referencing later, including the note that American goldfinches are notoriously unfaithful to any particular feeding spot which would explain their sudden and sporadic appearances, I've concluded that my backyard has now been discovered by both species of goldfinches.
It pays to observe birds carefully. What you at first believe to be one type of bird may actually be something very different, or you may discover that what you've been assuming were two different types of birds are actually different genders or ages of the same species. While some birds do change appearance more obviously with the seasons, even small changes can be worthwhile to note and will lead to a richer birding experience. Happy feeding!