One of the strongest predictors of whether or not an object in our environment captures our attention is the similarity between that object’s features and those we have prioritized—our attention set. For example, in inattentional blindness tasks, people are more likely to notice an unexpected white shape when tracking white objects than when tracking black objects. These attention sets are assumed to operate globally, enhancing perception of the critical feature across the entire visual field. We tested this assumption in four inattentional blindness experiments. Although observers were more likely to notice a white unexpected object when tracking white shapes, the effect of the participant’s attention set was substantially smaller or absent for objects appearing beyond the attended region. This result challenges the idea that attention sets help guide attention to relevant objects appearing in otherwise unattended regions.