Visual tracking is always a challenging problem due to many factors such as appearance changing, background clustering, illumination variations, occlusions and so on. Spatio-temporal context with the useful information can be applied as a valid approach to enhance the robustness of visual tracking. It can track scenes with partial occlusion and deformation, but it cannot have the capacity of coping with long-term occlusion. To overcome this limitation, we designed a long-term tracking framework based on spatio-temporal context. The proposed forward and backward tracking method utilizes spatial context information to realize long-term occlusion determination. Then, a cascade classifier consisting of a random fern classifier and a nearest neighbor classifier is constructed through on-line training after occlusion occurs to reposition the target to achieve long-term tracking.